How to use joblib.Parallel() to parallelize a loop with some "continue" in it?

user_10005931 0
Viewed 3542

Let's say I have a loop that "outputs" only if some condition is met :

output = []
for item in some_list:
    if condition(item):
        res = do_something(item)
        output.append(res)

For memory constraints I can't fill the output list with None-s, so I can't write :

joblib.Parallel(n_jobs=n)(joblib.delayed(do_something)(item) for item in some_list)

Is there a fast and reasonable way to make the list of outputs generated by Parallel not to include the redundant None-s?

EDIT

I forgot to clarify that both checking condition(item) and the actual do_something(item) take too long in aggregate so I'm trying to parallelize both.

joblib parallel-processing multiprocessing python
1 Answers

Q : "Is there a fast and reasonable way to make the list of outputs generated by Parallel not to include the redundant None-s?"

Yes.

Define the for item-generator part as needed :

joblib.Parallel( n_jobs = n )( joblib.delayed( do_something )( item )
                                                           for item in some_list
                                                 if condition( item )
                               )
user_36661970
Related