I have a DAG on Airflow (version: 1.10.15) which fails seemingly randomly, even though the code is not changed in between the different runs. It is a training script which normally should not take more than 2 hours, but occasionally it runs without ever completing, and eventually times out at the maximum set timeout.
When checking the logs, I'm seeing a continuous iteration of the following:
[2022-05-31 12:51:13,219] {databricks_operator.py:89} INFO - <SCRIPT_NAME> in run state: {'life_cycle_state':
'RUNNING', 'result_state': None, 'state_message': 'In run'}
[2022-05-31 12:51:13,219] {databricks_operator.py:90} INFO - View run status, Spark UI, and logs at <LOG_URL>
[2022-05-31 12:51:13,219] {databricks_operator.py:91} INFO - Sleeping for 30 seconds.
So there is seemingly no indication of failure in the logs. I then checked the actual notebook file, and saw that the execution of DAG stalls at the training step; one or two epochs are completed but then, no more training seems to occur until the timeout.
I'm pretty new to using Airflow, so at this point I'm not even sure where I should begin debugging this issue. If this was some error in the training script, it should give an error consistently, I presume. But in most cases the DAG actually runs correctly, it just stalls in an endless loop and times out only occasionally.
What might be going wrong?