my original script uses pool.map to run in parallel. I have logger setup in code to output to a file, and code running in different processes output logs to the same file.
now I tried to use dask for multiprocessing and found that the logger setup is not effective in other processes. my code can be simplified as below:
def setup_logger():
l = logging.getLogger()
....
return l
def fake_task(log: logging.Logger):
log.info('xxxxxxxx')
if __name__ == '__main__':
ll = setup_logger()
cluster = LocalCluster(n_workers=10, threads_per_worker=1)
client = Client(cluster)
tasks = []
for i in range(10):
tasks.append(dask.delayed(fake_task)(ll))
t = task.delayed(len)(tasks)
t.compute()
but I see nothing on console or in file. I tried to move setup_logger after client = Client(cluster), it still doesn't work.
I also tried something like logger = getLogger() and then use it direcly, still no luck
one option that I have is to use yaml configuration file. but this requires code changing and another issue is that the file name contains real time information.
for example, if I run my script at 08:30 AM,the filename will be something like 0830AM.log.I am not sure how to do this in configuration. and I also prefer to setup logger in code as it's already there.
another workaround I found is to call setup_logger in each fake_task method. but this is a little bit odd as I need to pass filename around.