I try to deploy my FastAPI application on AWS Fargate with an application load balancer in front of it. The application makes calls to AWS Aurora. My stack is as follows
- Python 3.7.4
- uvicorn 0.11.3
- gunicorn 19.9.0
- fastapi 0.53.2
- asyncpg 0.20.1
My container runs fine when deploying locally or on an EC2 instance, but workers time out during boot when deploying on Fargate. The logs don't give me any clue what is wrong (see below).
Things I tried to overcome this issue:
- increase memory for Fargate Task
- relax security group settings
- increase gunicorn timeout value
None of it had any effect.
However, when trying to deploy a barebone FastAPI application (just one root route, no dependencies) the application manages to boot and I can access it via the Load Balancer. Once I add my custom routes the issue occurs.
What other options do I have to debug this? If guincorn can't start my workers, is there a way to log the issue?
03:05:16 Checking for script in /app/prestart.sh
03:05:16 Running script /app/prestart.sh
03:05:16 Running inside /app/prestart.sh, you could add migrations to this file, e.g.:
03:05:16 #! /usr/bin/env bash
03:05:16 # Let the DB start
03:05:16 sleep 10;
03:05:16 # Run migrations
03:05:16 alembic upgrade head
03:05:17 [2020-04-03 03:05:17 +0000] [1] [DEBUG] Current configuration:
03:05:17 config: /app/app/gunicorn_conf.py
03:05:17 bind: ['0.0.0.0:80']
03:05:17 backlog: 2048
03:05:17 workers: 2
03:05:17 worker_class: uvicorn.workers.UvicornWorker
03:05:17 threads: 2
03:05:17 worker_connections: 1000
03:05:17 max_requests: 0
03:05:17 max_requests_jitter: 0
03:05:17 timeout: 120
03:05:17 graceful_timeout: 30
03:05:17 keepalive: 120
03:05:17 limit_request_line: 4094
03:05:17 limit_request_fields: 100
03:05:17 limit_request_field_size: 8190
03:05:17 reload: False
03:05:17 reload_engine: auto
03:05:17 reload_extra_files: []
03:05:17 spew: False
03:05:17 check_config: False
03:05:17 preload_app: False
03:05:17 sendfile: None
03:05:17 reuse_port: False
03:05:17 chdir: /app
03:05:17 daemon: False
03:05:17 raw_env: []
03:05:17 pidfile: None
03:05:17 worker_tmp_dir: None
03:05:17 user: 0
03:05:17 group: 0
03:05:17 umask: 0
03:05:17 initgroups: False
03:05:17 tmp_upload_dir: None
03:05:17 secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
03:05:17 forwarded_allow_ips: ['127.0.0.1']
03:05:17 accesslog: None
03:05:17 disable_redirect_access_to_syslog: False
03:05:17 access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
03:05:17 errorlog: -
03:05:17 loglevel: debug
03:05:17 capture_output: False
03:05:17 logger_class: gunicorn.glogging.Logger
03:05:17 logconfig: None
03:05:17 logconfig_dict: {}
03:05:17 syslog_addr: udp://localhost:514
03:05:17 syslog: False
03:05:17 syslog_prefix: None
03:05:17 syslog_facility: user
03:05:17 enable_stdio_inheritance: False
03:05:17 statsd_host: None
03:05:17 statsd_prefix:
03:05:17 proc_name: None
03:05:17 default_proc_name: app.main:app
03:05:17 pythonpath: None
03:05:17 paste: None
03:05:17 on_starting: <function OnStarting.on_starting at 0x7ff9c4742ef0>
03:05:17 on_reload: <function OnReload.on_reload at 0x7ff9c4757050>
03:05:17 when_ready: <function WhenReady.when_ready at 0x7ff9c4757170>
03:05:17 pre_fork: <function Prefork.pre_fork at 0x7ff9c4757290>
03:05:17 post_fork: <function Postfork.post_fork at 0x7ff9c47573b0>
03:05:17 post_worker_init: <function PostWorkerInit.post_worker_init at 0x7ff9c47574d0>
03:05:17 worker_int: <function WorkerInt.worker_int at 0x7ff9c47575f0>
03:05:17 worker_abort: <function WorkerAbort.worker_abort at 0x7ff9c4757710>
03:05:17 pre_exec: <function PreExec.pre_exec at 0x7ff9c4757830>
03:05:17 pre_request: <function PreRequest.pre_request at 0x7ff9c4757950>
03:05:17 post_request: <function PostRequest.post_request at 0x7ff9c47579e0>
03:05:17 child_exit: <function ChildExit.child_exit at 0x7ff9c4757b00>
03:05:17 worker_exit: <function WorkerExit.worker_exit at 0x7ff9c4757c20>
03:05:17 nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7ff9c4757d40>
03:05:17 on_exit: <function OnExit.on_exit at 0x7ff9c4757e60>
03:05:17 proxy_protocol: False
03:05:17 proxy_allow_ips: ['127.0.0.1']
03:05:17 keyfile: None
03:05:17 certfile: None
03:05:17 ssl_version: 2
03:05:17 cert_reqs: 0
03:05:17 ca_certs: None
03:05:17 suppress_ragged_eofs: True
03:05:17 do_handshake_on_connect: False
03:05:17 ciphers: TLSv1
03:05:17 raw_paste_global_conf: []
03:05:17 [2020-04-03 03:05:17 +0000] [1] [INFO] Starting gunicorn 19.9.0
03:05:17 [2020-04-03 03:05:17 +0000] [1] [DEBUG] Arbiter booted
03:05:17 [2020-04-03 03:05:17 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)
03:05:17 [2020-04-03 03:05:17 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
03:05:17 [2020-04-03 03:05:17 +0000] [8] [INFO] Booting worker with pid: 8
03:05:17 [2020-04-03 03:05:17 +0000] [9] [INFO] Booting worker with pid: 9
03:05:17 [2020-04-03 03:05:17 +0000] [1] [DEBUG] 2 workers
03:07:17 [2020-04-03 03:07:17 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:8)
03:07:17 [2020-04-03 03:07:17 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:9)
03:07:17 [2020-04-03 03:07:17 +0000] [12] [INFO] Booting worker with pid: 12
03:07:18 [2020-04-03 03:07:18 +0000] [13] [INFO] Booting worker with pid: 13
03:07:54 [2020-04-03 03:07:54 +0000] [1] [INFO] Handling signal: term