How to dynamically combine generators?

user_15065346 0
Viewed 63
for submissions in itertools.zip_longest(submission_stream, submission_stream2): #want to put all streams here
            for submission in submissions:
                # processing

The above code works for two streams that I have initialised. My goal is to combine streams based on username in a .csv file. If a username is there, run a stream for them. If it gets removed, or a new username is added, remove or start that stream respectively. An example of a stream is:

submission_stream = reddit.redditor("username").stream.submissions(skip_existing=True, pause_after=-1)

I would really appreciate if someone would guide me.

praw reddit itertools generator iterator
1 Answers

You would probably have to start streaming over again every time your .csv file is changed, although you could get away with filtering (itertools.filterfalse) for username removals. Code sketch, assuming functions to get a list of streams, determine if a submission belongs to deleted username, and determine if file was changed with an addition:

while True:
    streams = get_list_of_streams_from_csv()
    for submissions in itertools.zip_longest(streams):
        for submission in itertools.filterfalse(not_deleted, submissions):
            #processing
        if csv_changed_to_add()
            break

Adding in additional streams, capturing deletion with .filterfalse:

streams = get_list_of_streams_from_csv()
zip_iter = itertools.zip_longest(streams)
while True:
    for submission in itertools.filterfalse(not_deleted, zip_iter):
        #processing
        if csv_changed_to_add()
            break    
    zip_iter = itertools.zip_longest(zip_iter, get_list_of_new_streams())
user_56603150
Related