PyTorch: Logging during model.fit()

Dávid Gajdoš 0
Viewed 17

I'm using pytorch/fastai for training models. Since I'm working with remote machines, I am running the scripts using nohup python $1 >$2 2>&1 & with redirection to logging file like "log123.txt".

My problem is that during the model.fit() phase with scheduler, I can't see the progress in the file after each epoch like in console and the results are written to my logging file after the model.fit() finishes training. It works fine when I watch the process in console.

Is there any workaround for this?

fast-ai pytorch python
1 Answers

This may be due to file buffering. You can disable it for python process like this:

  1. Run your script with -u options. Example:

python -u my_script.py > log.txt

  1. Use PYTHONUNBUFFERED:

PYTHONUNBUFFERED=1 python my_script.py > log.txt

or

export PYTHONUNBUFFERED=1
python my_script.py > log.txt
u1234x12340
Related