I have a situation in which I want to divide an I/O bound task based on some parameters. In that task, I want to do further division based on some other calculated parameters. Let me try to illustrate the scenario via some basic code:
def inner_task(a, b, c, d):
some_result = doWork(a, b, c, d)
return some_result
def outer_task(a, b):
c, d = SomeCalculations()
with ThreadPoolExecutor(max_workers=4) as inner_pool:
inner_pool.submit(inner_task, a, b, c, d)
if __name__ == '__main__':
with ThreadPoolExecutor(max_workers=10) as outer_pool:
outer_pool.submit(outer_task, a, b)
My question is that, whether it is safe to do something like that (basically a thread pool inside another pool) both from a safety and performance point of view?.
P.S. I know that I can simply create one pool with 10 x 4 workers and assign jobs to them without dividing them into two tasks but, in this way my implementation is encapsulated much better and everything stays nicely in its own place.
Thanks for your time to review my scenario.