I have an external service that calls the airflow trigger DAG API (quite alot of requests) at once during a fixed timing.
I am having Gunicorn timeout despite tweaking the airflow.cfg settings. master_timeout, worker_timeout, and refresh_interval. I am also using gevent worker type for async.
Is there something am missing out?
# Number of seconds the webserver waits before killing gunicorn master that doesn't respond
web_server_master_timeout = 600
# Number of seconds the gunicorn webserver waits before timing out on a worker
web_server_worker_timeout = 480
# Number of workers to refresh at a time. When set to 0, worker refresh is
# disabled. When nonzero, airflow periodically refreshes webserver workers by
# bringing up new ones and killing old ones.
worker_refresh_batch_size = 1
# Number of seconds to wait before refreshing a batch of workers.
worker_refresh_interval = 30
# Number of workers to run the Gunicorn web server
workers = 5
# The worker class gunicorn should use. Choices include
# sync (default), eventlet, gevent
worker_class = gevent