I'll need a bit of help here.
running airflow on docker container+LocalExecutor.
Airflow version is 2.0.0 (https://pypi.org/project/apache-airflow/2.0.0/)
I'm running a long runnning task with a wrapper of SSHOperator. Basically I open an SSH Session to run a spark-submit job in spark edge node. (YARN JOB succeeds but airflow task fails)
Task starts with PID 31675:
[2021-06-24 18:29:09,664] {standard_task_runner.py:51} INFO - Started process 31675 to run task
Then after sometime getting this warning:
Recorded pid 1098 does not match the current pid 31631
And then the tasks fails:
[2021-06-24 19:45:44,493] {local_task_job.py:166} WARNING - Recorded pid 1098 does not match the current pid 31631
[2021-06-24 19:45:44,496] {process_utils.py:95} INFO - Sending Signals.SIGTERM to GPID 31675
[2021-06-24 19:45:44,496] {taskinstance.py:1214} ERROR - Received SIGTERM. Terminating subprocesses.
[2021-06-24 19:45:44,528] {taskinstance.py:1396} ERROR - LatamSSH operator error: Task received SIGTERM signal
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/latamairflow/operators/latam_ssh_operator.py", line 453, in execute
readq, _, _ = select([channel], [], [], self.timeout)
File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1216, in signal_handler
raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1086, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1300, in _execute_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.6/site-packages/latamairflow/operators/latam_ssh_operator.py", line 502, in execute
raise AirflowException("LatamSSH operator error: {0}".format(str(e)))
airflow.exceptions.AirflowException: LatamSSH operator error: Task received SIGTERM signal
[2021-06-24 19:45:44,529] {taskinstance.py:1440} INFO - Marking task as FAILED.