how to make python logger setup effective in dask

Viewed 23

my original script uses pool.map to run in parallel. I have logger setup in code to output to a file, and code running in different processes output logs to the same file.

now I tried to use dask for multiprocessing and found that the logger setup is not effective in other processes. my code can be simplified as below:

def setup_logger():
   l = logging.getLogger()
   ....
   return l

def fake_task(log: logging.Logger):
    log.info('xxxxxxxx')


if __name__ == '__main__':
    ll = setup_logger()
    cluster = LocalCluster(n_workers=10, threads_per_worker=1)
    client = Client(cluster)
    tasks = []
    for i in range(10):
        tasks.append(dask.delayed(fake_task)(ll))
    t = task.delayed(len)(tasks)
    t.compute()

but I see nothing on console or in file. I tried to move setup_logger after client = Client(cluster), it still doesn't work.

I also tried something like logger = getLogger() and then use it direcly, still no luck

one option that I have is to use yaml configuration file. but this requires code changing and another issue is that the file name contains real time information.

for example, if I run my script at 08:30 AM,the filename will be something like 0830AM.log.I am not sure how to do this in configuration. and I also prefer to setup logger in code as it's already there.

another workaround I found is to call setup_logger in each fake_task method. but this is a little bit odd as I need to pass filename around.

0 Answers
Related