Brief
Django-Oscar web application running in production and staging environment, hosted in virtual environment, running as systemd daemon using Gunicorn.
Nginx is used as proxy, gunicorn is working normally left to the default "worker-class" which is "sync", but its not suitable for our case, I need to change it to async, specifically "gevent", usually its done by adding the following parameters to ExecStart
--worker-class gevent --worker-connections 768
But the workers did not start and I got the following error
[2017-08-15 08:42:35 +0000] [25425] [INFO] Starting gunicorn 19.7.1
[2017-08-15 08:42:35 +0000] [25425] [DEBUG] Arbiter booted
[2017-08-15 08:42:35 +0000] [25425] [INFO] Listening at: unix:/tmp/myapp.sock (25425)
[2017-08-15 08:42:35 +0000] [25425] [INFO] Using worker: gevent
[2017-08-15 08:42:35 +0000] [25430] [INFO] Booting worker with pid: 25430
[2017-08-15 08:42:35 +0000] [25425] [DEBUG] 1 workers
[2017-08-15 11:42:37 +0000] [25430] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
worker.init_process()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 190, in init_process
super(GeventWorker, self).init_process()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
self.load_wsgi()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 135, in load_wsgi
self.wsgi = self.app.wsgi()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/util.py", line 352, in import_app
__import__(module)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
result = _import(*args, **kwargs)
File "/home/ubuntu/myapp/myapp/wsgi.py", line 16, in <module>
application = get_wsgi_application()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/django/core/wsgi.py", line 14, in get_wsgi_application
django.setup()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
apps.populate(settings.INSTALLED_APPS)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
app_config.import_models(all_models)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/django/apps/config.py", line 198, in import_models
self.models_module = import_module(models_module_name)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
result = _import(*args, **kwargs)
File "/home/ubuntu/myapp/main/checkout/models.py", line 5, in <module>
from main.shipping.models import ShippingPeriod
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
result = _import(*args, **kwargs)
File "/home/ubuntu/myapp/main/shipping/models.py", line 57, in <module>
from oscar.apps.shipping.models import * # noqa
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
result = _import(*args, **kwargs)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/oscar/apps/shipping/models.py", line 1, in <module>
from oscar.apps.shipping import abstract_models
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
result = _import(*args, **kwargs)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/oscar/apps/shipping/abstract_models.py", line 12, in <module>
Scale = loading.get_class('shipping.scales', 'Scale')
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/oscar/core/loading.py", line 29, in get_class
return get_classes(module_label, [classname], module_prefix)[0]
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/oscar/core/loading.py", line 100, in get_classes
local_module = _import_module(local_module_label, classnames)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/oscar/core/loading.py", line 121, in _import_module
return __import__(module_label, fromlist=classnames)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
result = _import(*args, **kwargs)
ImportError: No module named scales
[2017-08-15 11:42:37 +0000] [25430] [INFO] Worker exiting (pid: 25430)
[2017-08-15 08:42:37 +0000] [25425] [INFO] Shutting down: Master
[2017-08-15 08:42:37 +0000] [25425] [INFO] Reason: Worker failed to boot.
After reading and trying many solutions, I tried --preload in the mix, and it got the workers up, but unfortunately for some reason its causing the application to return internal server error (500)
[2017-08-15 14:44:51 +0000] [26323] [DEBUG] GET /
[2017-08-15 14:44:51 +0000] [26323] [ERROR] Error handling request /
Traceback (most recent call last):
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/workers/async.py", line 56, in handle
self.handle_request(listener_name, req, client, addr)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 152, in handle_request
super(GeventWorker, self).handle_request(*args)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/workers/async.py", line 107, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 170, in __call__
self.load_middleware()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 50, in load_middleware
mw_class = import_string(middleware_path)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/django/utils/module_loading.py", line 26, in import_string
module = import_module(module_path)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
result = _import(*args, **kwargs)
File "/home/ubuntu/myapp/oscarapi/middleware.py", line 16, in <module>
from oscarapi.basket.operations import (
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
result = _import(*args, **kwargs)
File "/home/ubuntu/myapp/oscarapi/basket/operations.py", line 22, in <module>
Applicator = get_class('offer.applicator', 'Applicator')
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/oscar/core/loading.py", line 29, in get_class
return get_classes(module_label, [classname], module_prefix)[0]
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/oscar/core/loading.py", line 100, in get_classes
local_module = _import_module(local_module_label, classnames)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/oscar/core/loading.py", line 121, in _import_module
return __import__(module_label, fromlist=classnames)
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
result = _import(*args, **kwargs)
ImportError: No module named applicator
==> /var/log/nginx/access.log <==
10.0.52.154 - - [15/Aug/2017:11:44:51 +0000] "GET / HTTP/1.1" 500 141 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"
Can someone kindly point me in the right direction :) Thanks in advance everyone!
Update I also tried eventlet since gevent is not working, and I get this
[2017-08-16 07:19:30 +0000] [7481] [DEBUG] Current configuration:
proxy_protocol: False
worker_connections: 768
statsd_host: None
max_requests_jitter: 0
post_fork: <function post_fork at 0x7f1637e18938>
errorlog: /tmp/logging.log
enable_stdio_inheritance: False
worker_class: eventlet
ssl_version: 2
suppress_ragged_eofs: True
syslog: False
syslog_facility: user
when_ready: <function when_ready at 0x7f1637e18668>
pre_fork: <function pre_fork at 0x7f1637e187d0>
cert_reqs: 0
preload_app: False
keepalive: 2
accesslog: None
group: 1000
graceful_timeout: 30
do_handshake_on_connect: False
spew: False
workers: 3
proc_name: None
sendfile: None
pidfile: None
umask: 0
on_reload: <function on_reload at 0x7f1637e18500>
pre_exec: <function pre_exec at 0x7f1637e18ed8>
worker_tmp_dir: None
limit_request_fields: 100
pythonpath: None
on_exit: <function on_exit at 0x7f1637e1e758>
config: None
logconfig: None
check_config: False
statsd_prefix:
secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
reload_engine: auto
proxy_allow_ips: ['127.0.0.1']
pre_request: <function pre_request at 0x7f1637e1e0c8>
post_request: <function post_request at 0x7f1637e1e1b8>
forwarded_allow_ips: ['127.0.0.1']
worker_int: <function worker_int at 0x7f1637e18c08>
raw_paste_global_conf: []
threads: 1
max_requests: 0
chdir: /home/ubuntu
daemon: False
user: 1000
limit_request_line: 4094
access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
certfile: None
on_starting: <function on_starting at 0x7f1637e18398>
post_worker_init: <function post_worker_init at 0x7f1637e18aa0>
child_exit: <function child_exit at 0x7f1637e1e320>
worker_exit: <function worker_exit at 0x7f1637e1e488>
paste: None
default_proc_name: myapp.wsgi:application
syslog_addr: udp://localhost:514
syslog_prefix: None
ciphers: TLSv1
worker_abort: <function worker_abort at 0x7f1637e18d70>
loglevel: debug
bind: ['unix:/tmp/myapp.sock']
raw_env: []
initgroups: False
capture_output: False
reload: False
limit_request_field_size: 8190
nworkers_changed: <function nworkers_changed at 0x7f1637e1e5f0>
timeout: 30
keyfile: None
ca_certs: None
tmp_upload_dir: None
backlog: 2048
logger_class: gunicorn.glogging.Logger
[2017-08-16 07:19:30 +0000] [7481] [INFO] Starting gunicorn 19.7.1
[2017-08-16 07:19:30 +0000] [7481] [DEBUG] Arbiter booted
[2017-08-16 07:19:30 +0000] [7481] [INFO] Listening at: unix:/tmp/myapp.sock (7481)
[2017-08-16 07:19:30 +0000] [7481] [INFO] Using worker: eventlet
[2017-08-16 07:19:30 +0000] [7486] [INFO] Booting worker with pid: 7486
[2017-08-16 07:19:30 +0000] [7487] [INFO] Booting worker with pid: 7487
[2017-08-16 07:19:30 +0000] [7487] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
worker.init_process()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/workers/geventlet.py", line 102, in init_process
self.patch()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/workers/geventlet.py", line 91, in patch
hubs.use_hub()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 70, in use_hub
mod = get_default_hub()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 38, in get_default_hub
import eventlet.hubs.epolls
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 27, in <module>
from eventlet.hubs.hub import BaseHub
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 23, in <module>
from eventlet.support import greenlets as greenlet, clear_sys_exc_info, monotonic, six
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/eventlet/support/monotonic.py", line 167, in <module>
raise RuntimeError('no suitable implementation for this system')
RuntimeError: no suitable implementation for this system
[2017-08-16 07:19:30 +0000] [7486] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
worker.init_process()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/workers/geventlet.py", line 102, in init_process
self.patch()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/gunicorn/workers/geventlet.py", line 91, in patch
hubs.use_hub()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 70, in use_hub
mod = get_default_hub()
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 38, in get_default_hub
import eventlet.hubs.epolls
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 27, in <module>
from eventlet.hubs.hub import BaseHub
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 23, in <module>
from eventlet.support import greenlets as greenlet, clear_sys_exc_info, monotonic, six
File "/home/ubuntu/.virtualenv/myapp-env/local/lib/python2.7/site-packages/eventlet/support/monotonic.py", line 167, in <module>
raise RuntimeError('no suitable implementation for this system')
RuntimeError: no suitable implementation for this system
[2017-08-16 07:19:30 +0000] [7486] [INFO] Worker exiting (pid: 7486)
[2017-08-16 07:19:30 +0000] [7487] [INFO] Worker exiting (pid: 7487)
[2017-08-16 07:19:30 +0000] [7481] [INFO] Shutting down: Master
[2017-08-16 07:19:30 +0000] [7481] [INFO] Reason: Worker failed to boot.