I have a Python/Flask app running on Gunicorn. For some reason, somewhere between 2 to 2.5 hours of starting the app, this always happens:
[2021-05-19 03:17:38 +0000] [26530] [DEBUG] Closing connection.
[2021-05-19 03:18:04 +0000] [26526] [WARNING] Worker with pid 26529 was terminated due to signal 1
[2021-05-19 03:18:04 +0000] [26526] [WARNING] Worker with pid 26530 was terminated due to signal 1
[2021-05-19 03:18:04 +0000] [26526] [WARNING] Worker with pid 26527 was terminated due to signal 1
[2021-05-19 03:18:04 +0000] [26526] [WARNING] Worker with pid 26528 was terminated due to signal 1
[2021-05-19 03:18:04 +0000] [27837] [INFO] Booting worker with pid: 27837
[2021-05-19 03:18:04 +0000] [26526] [DEBUG] 1 workers
[2021-05-19 03:18:04 +0000] [26526] [INFO] Handling signal: hup
[2021-05-19 03:18:04 +0000] [26526] [INFO] Hang up: Master
[2021-05-19 03:18:04 +0000] [26526] [DEBUG] Current configuration:
config: ./gunicorn.conf.py
wsgi_app: None
bind: ['0.0.0.0:8081']
backlog: 2048
workers: 4
worker_class: sync
threads: 1
worker_connections: 1000
max_requests: 0
max_requests_jitter: 0
timeout: 9223372036
graceful_timeout: 30
keepalive: 2
limit_request_line: 4094
limit_request_fields: 100
limit_request_field_size: 8190
reload: False
reload_engine: auto
reload_extra_files: []
spew: False
check_config: False
print_config: False
preload_app: False
sendfile: None
reuse_port: False
chdir: /home/ubuntu
daemon: False
raw_env: []
pidfile: None
worker_tmp_dir: None
user: 0
group: 0
umask: 0
initgroups: False
tmp_upload_dir: None
secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
forwarded_allow_ips: ['127.0.0.1']
accesslog: None
disable_redirect_access_to_syslog: False
access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
errorlog: -
loglevel: debug
capture_output: False
logger_class: gunicorn.glogging.Logger
logconfig: None
logconfig_dict: {}
syslog_addr: udp://localhost:514
syslog: False
syslog_prefix: None
syslog_facility: user
enable_stdio_inheritance: False
statsd_host: None
dogstatsd_tags:
statsd_prefix:
proc_name: None
default_proc_name: __init__:create_app()
pythonpath: None
paste: None
on_starting: <function OnStarting.on_starting at 0x7f3f9ecc3ca0>
on_reload: <function OnReload.on_reload at 0x7f3f9ecc3dc0>
when_ready: <function WhenReady.when_ready at 0x7f3f9ecc3ee0>
pre_fork: <function Prefork.pre_fork at 0x7f3f9ec5a040>
post_fork: <function Postfork.post_fork at 0x7f3f9ec5a160>
post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f3f9ec5a280>
worker_int: <function WorkerInt.worker_int at 0x7f3f9ec5a3a0>
worker_abort: <function WorkerAbort.worker_abort at 0x7f3f9ec5a4c0>
pre_exec: <function PreExec.pre_exec at 0x7f3f9ec5a5e0>
pre_request: <function PreRequest.pre_request at 0x7f3f9ec5a700>
post_request: <function PostRequest.post_request at 0x7f3f9ec5a790>
child_exit: <function ChildExit.child_exit at 0x7f3f9ec5a8b0>
worker_exit: <function WorkerExit.worker_exit at 0x7f3f9ec5a9d0>
nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f3f9ec5aaf0>
on_exit: <function OnExit.on_exit at 0x7f3f9ec5ac10>
proxy_protocol: False
proxy_allow_ips: ['127.0.0.1']
keyfile: /etc/letsencrypt/live/site.io/privkey.pem
certfile: /etc/letsencrypt/live/site.io/fullchain.pem
ssl_version: 2
cert_reqs: 0
ca_certs: None
suppress_ragged_eofs: True
do_handshake_on_connect: False
ciphers: None
raw_paste_global_conf: []
strip_header_spaces: False
[2021-05-19 03:18:04 +0000] [26526] [DEBUG] 4 workers
[2021-05-19 03:18:04 +0000] [27838] [INFO] Booting worker with pid: 27838
[2021-05-19 03:18:04 +0000] [27839] [INFO] Booting worker with pid: 27839
[2021-05-19 03:18:04 +0000] [27841] [INFO] Booting worker with pid: 27841
[2021-05-19 03:18:04 +0000] [27840] [INFO] Booting worker with pid: 27840
[2021-05-19 03:18:08 +0000] [27837] [INFO] Worker exiting (pid: 27837)
[2021-05-19 03:18:08 +0000] [26526] [WARNING] Worker with pid 27837 was terminated due to signal 15
I have no idea how or why Gunicorn is apparently receiving a HUP signal (https://docs.gunicorn.org/en/19.3/signals.html#reload-the-configuration) causing it to reload the app. Often nothing else is happening in the app - no users using it or anything else in the logs. This is a problem since if there is someone using the app, restarting it in the middle of some code running (there's a function that can take minutes to finish) will break it.
I'm running an AWS EC2 t2.medium instance, python3.9 and Gunicorn 20.1.0, launching the app as follows:
/usr/bin/python3.9 -m gunicorn.app.wsgiapp -w 4 -b 0.0.0.0:8081 -t 9223372036 '__init__:create_app()' --certfile=/etc/letsencrypt/live/site.io/fullchain.pem --keyfile=/etc/letsencrypt/live/site.io/privkey.pem --log-level 'debug'
How can I stop Gunicorn restarting like this?