I want to save the standard out from my dag separately from the logging that airflow generates.
My scripts print to standard out and I want to capture that (and only that) into a specified log file each time it runs. I've tried
logfile = 'mylog_01.log'
myfunc = def()...
log = open(logfile, "ab")
sys.stdout = log
print_params = PythonOperator(task_id="print_params",
python_callable=myfunc,
provide_context=True,
dag=dag)
... but std.out is reset within each PythonOperator. How do I control where standard out is printing to within all python operators?
Also, I am using hundreds of print statements within the scripts so modifying each of those isn't practical in this case.