I've made a DAG with an "unzip" task, but every time I run it the task fail (timeout) silently without giving any output.
For context the previous tasks are download tasks and take more then an hour, they never fail/skip/timeout they always finish, but when the unzip task start it timeout after around 10 minutes each times. I tried to set many different values in the config to lengthen the timeout but nothing seems to work. I've check for configuration to change the timeout time:
[core]
executor = CeleryExecutor
[celery]
task_adoption_timeout = 6000
operation_timeout = 6000
The airflow instance is deployed on GCP's Composer.
When I enable retry I can see the time of timeout clearly because the last log (just before the unzip start) is almost always ~10 minutes apart from the next log (of the retry task starting). I couldn't find any variable changing this timeout in the Airflow doc, apart from task_adoption_timeout which doesn't seem to do anything.
the task:
with zipfile.ZipFile(zip_file_path, "r") as z:
z.extractall()
What I wan't is either for the timeout to be long enough for the unzip to processed (not the best option) OR a way to unzip in Python without causing Airflow to think I'm timing out.
Btw I read other stackoverflow answers but none worked: Apache Airflow Task timeout