I have this Airflow Task that must run docker-compose.
The command below works perfectly well alone as the airflow user.
docker-compose -f /home/airflow/update-scraping/docker-compose.yml up --build
But when I wrap it in a BashOperator I got this error :
[2021-01-27 23:32:13,903] {taskinstance.py:901} INFO - Executing <Task(BashOperator): update-scraping> on 2021-01-27T23:27:05.828047+00:00
[2021-01-27 23:32:13,905] {standard_task_runner.py:54} INFO - Started process 111626 to run task
[2021-01-27 23:32:13,944] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'update-scraping', 'update-scraping', '2021-01-27T23:27:05.828047+00:00', '--job_id', '99', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/update-scraping.py', '--cfg_path', '/tmp/tmpoo___wh5']
[2021-01-27 23:32:13,945] {standard_task_runner.py:78} INFO - Job 99: Subtask update-scraping
[2021-01-27 23:32:13,996] {logging_mixin.py:112} INFO - Running <TaskInstance: update-scraping.update-scraping 2021-01-27T23:27:05.828047+00:00 [running]> on host ip-10-0-149-233.eu-west-3.compute.internal
[2021-01-27 23:32:14,071] {bash_operator.py:113} INFO - Tmp dir root location:
/tmp
[2021-01-27 23:32:14,072] {bash_operator.py:134} INFO - Temporary script location: /tmp/airflowtmpxdo3epew/update-scrapingsmc6tuxa
[2021-01-27 23:32:14,072] {bash_operator.py:146} INFO - Running command: docker-compose -f /home/airflow/update-scraping/docker-compose.yml up --build
[2021-01-27 23:32:14,079] {bash_operator.py:153} INFO - Output:
[2021-01-27 23:32:14,619] {bash_operator.py:157} INFO - [111632] Failed to execute script docker-compose
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - Traceback (most recent call last):
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "urllib3/connectionpool.py", line 677, in urlopen
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "urllib3/connectionpool.py", line 392, in _make_request
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "http/client.py", line 1277, in request
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "http/client.py", line 1323, in _send_request
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "http/client.py", line 1272, in endheaders
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "http/client.py", line 1032, in _send_output
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "http/client.py", line 972, in send
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "docker/transport/unixconn.py", line 43, in connect
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - PermissionError: [Errno 13] Permission denied
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO -
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - During handling of the above exception, another exception occurred:
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO -
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - Traceback (most recent call last):
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "requests/adapters.py", line 449, in send
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "urllib3/connectionpool.py", line 727, in urlopen
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "urllib3/util/retry.py", line 410, in increment
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "urllib3/packages/six.py", line 734, in reraise
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "urllib3/connectionpool.py", line 677, in urlopen
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "urllib3/connectionpool.py", line 392, in _make_request
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "http/client.py", line 1277, in request
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "http/client.py", line 1323, in _send_request
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "http/client.py", line 1272, in endheaders
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "http/client.py", line 1032, in _send_output
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "http/client.py", line 972, in send
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - File "docker/transport/unixconn.py", line 43, in connect
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO -
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO - During handling of the above exception, another exception occurred:
[2021-01-27 23:32:14,620] {bash_operator.py:157} INFO -
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - Traceback (most recent call last):
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "docker/api/client.py", line 214, in _retrieve_server_version
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "docker/api/daemon.py", line 181, in version
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "docker/utils/decorators.py", line 46, in inner
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "docker/api/client.py", line 237, in _get
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "requests/sessions.py", line 543, in get
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "requests/sessions.py", line 530, in request
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "requests/sessions.py", line 643, in send
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "requests/adapters.py", line 498, in send
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO -
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - During handling of the above exception, another exception occurred:
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO -
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - Traceback (most recent call last):
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "docker-compose", line 3, in <module>
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "compose/cli/main.py", line 80, in main
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "compose/cli/main.py", line 189, in perform_command
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "compose/cli/command.py", line 70, in project_from_options
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "compose/cli/command.py", line 153, in get_project
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "compose/cli/docker_client.py", line 43, in get_client
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "compose/cli/docker_client.py", line 170, in docker_client
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "docker/api/client.py", line 197, in __init__
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - File "docker/api/client.py", line 222, in _retrieve_server_version
[2021-01-27 23:32:14,621] {bash_operator.py:157} INFO - docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
[2021-01-27 23:32:14,702] {bash_operator.py:159} INFO - Command exited with return code 255
[2021-01-27 23:32:14,718] {taskinstance.py:1150} ERROR - Bash command failed
Traceback (most recent call last):
File "/home/airflow/miniconda3/envs/airflow/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/airflow/miniconda3/envs/airflow/lib/python3.9/site-packages/airflow/operators/bash_operator.py", line 165, in execute
raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
[2021-01-27 23:32:14,720] {taskinstance.py:1187} INFO - Marking task as UP_FOR_RETRY. dag_id=update-scraping, task_id=update-scraping, execution_date=20210127T232705, start_date=20210127T233213, end_date=20210127T233214
[2021-01-27 23:32:18,854] {local_task_job.py:102} INFO - Task exited with return code 1