I have 2 airflow workers, each in a machine In machine A a file sensor task runs, while in machine B an rsync task (copies files from A to B) and also another file sensor task for the copied files. Sometimes rsync task fails without trying to execute and does not produce a log file. The trigger rule of tasks is all_done and airflow v1.9
The Sensor Tasks use xcom_push and xcom_pull which I disabled but the same issue happens
-[ RECORD 3 ]---+-----------------------------------------------------
task_id | task_name (rsync)
dag_id | dag_name
execution_date | 2019-10-08 19:30:00
start_date |
end_date | 2019-10-08 20:20:21.649662
duration |
state | failed
try_number | 0
hostname |
unixname | nio
job_id |
pool | pool_name
queue | queue name of machine B
priority_weight | 3
operator |
queued_dttm | 2019-10-08 20:20:20.550397
pid |
max_tries | 3
operator is null retry is 0 and the state is failed
Expect that the rsync task will run.