My project recently added django channel for websocket and layer:
Django=3.2.6
Channel=3.0.4
channels-redis="3.3.0"
gevent=21.8.0
gunicorn=20.1.0
Gunicorn and daphne run WSGI and ASGI separately, Nginx proxy the traffic to WSGI or ASGI. It is part of supervisord config
[program:myproj]
command=/usr/local/bin/gunicorn -c gunicorn_conf.py myproj.wsgi
[fcgi-program:myproj_channel]
socket=tcp://0.0.0.0:8002
directory=/opt/www/myproj
command=daphne -u /var/run/daphne/daphne%(process_num)d.sock --fd 0 --access-log - --proxy-headers myproj.asgi:application
In my production environment, it has less 0.1% chances to get cannot call this from an async context - use a thread or sync_to_async. These errors are not from ASGI(daphne) but from normal WSGI(Gunicorn) requests, they didn't occur in same piece of code but the root cause is a database query.
For example, this is in a database query:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/usr/local/lib/python3.8/dist-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.8/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/django/views/generic/base.py", line 70, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/usr/local/lib/python3.8/dist-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/usr/local/lib/python3.8/dist-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/usr/local/lib/python3.8/dist-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/opt/www/myproj/myproj/services/member_service.py", line 275, in _wrapped_view
return view_func(api_view, request, *args, **kwargs)
File "/opt/www/myproj/myproj/api/restful/form.py", line 108, in get
published_form = PublishedForm.objects.get_latest_published_for_form(form_identifier, try_master_form=False)
File "/opt/www/myproj/myproj/webform/models.py", line 44, in get_latest_published_for_form
return self.filter(form_case__identifier=form_identifier, form_case__form_state=FORM_STATE_PUBLISHED).latest('version')
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 670, in latest
return self.reverse()._earliest(*fields)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 664, in _earliest
return obj.get()
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 431, in get
num = len(clone)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 262, in __len__
self._fetch_all()
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 1324, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 51, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
cursor = self.connection.cursor()
File "/usr/local/lib/python3.8/dist-packages/django/utils/asyncio.py", line 24, in inner
raise SynchronousOnlyOperation(message)
This is on session save
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/usr/local/lib/python3.8/dist-packages/django/utils/deprecation.py", line 119, in __call__
response = self.process_response(request, response)
File "/usr/local/lib/python3.8/dist-packages/django/contrib/sessions/middleware.py", line 61, in process_response
request.session.save()
File "/usr/local/lib/python3.8/dist-packages/django/contrib/sessions/backends/db.py", line 81, in save
return self.create()
File "/usr/local/lib/python3.8/dist-packages/django/contrib/sessions/backends/db.py", line 51, in create
self._session_key = self._get_new_session_key()
File "/usr/local/lib/python3.8/dist-packages/django/contrib/sessions/backends/base.py", line 196, in _get_new_session_key
if not self.exists(session_key):
File "/usr/local/lib/python3.8/dist-packages/django/contrib/sessions/backends/db.py", line 47, in exists
return self.model.objects.filter(session_key=session_key).exists()
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 808, in exists
return self.query.has_results(using=self.db)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/sql/query.py", line 552, in has_results
return compiler.has_results()
File "/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py", line 1145, in has_results
return bool(self.execute_sql(SINGLE))
File "/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
cursor = self.connection.cursor()
File "/usr/local/lib/python3.8/dist-packages/django/utils/asyncio.py", line 24, in inner
raise SynchronousOnlyOperation(message)
This is at session read:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/django/contrib/sessions/backends/base.py", line 233, in _get_session
return self._session_cache
During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:
File "/usr/local/lib/python3.8/dist-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/opt/www/myproj/myproj/common/middleware.py", line 24, in __call__
if 'prefer_locale' not in request.session:
File "/usr/local/lib/python3.8/dist-packages/django/contrib/sessions/backends/base.py", line 55, in __contains__
return key in self._session
File "/usr/local/lib/python3.8/dist-packages/django/contrib/sessions/backends/base.py", line 238, in _get_session
self._session_cache = self.load()
File "/usr/local/lib/python3.8/dist-packages/django/contrib/sessions/backends/db.py", line 43, in load
s = self._get_session_from_db()
File "/usr/local/lib/python3.8/dist-packages/django/contrib/sessions/backends/db.py", line 32, in _get_session_from_db
return self.model.objects.get(
File "/usr/local/lib/python3.8/dist-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 431, in get
num = len(clone)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 262, in __len__
self._fetch_all()
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 1324, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 51, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
cursor = self.connection.cursor()
File "/usr/local/lib/python3.8/dist-packages/django/utils/asyncio.py", line 24, in inner
raise SynchronousOnlyOperation(message)
My project doesn't have middleware special for channel - my project runs for years, just recently added channel so no many customised settings for channel.
Installed Middleware:
['myproj.common.middleware.SameSiteMiddleware',
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'corsheaders.middleware.CorsMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django_otp.middleware.OTPMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'myproj.common.middleware.MembershipMiddleware',
'myproj.common.middleware.AutoLogoutMiddleware']
I am pretty sure these part codes don't have any async methods. Because except websocket code, I only found one place using asyncio (that also in celery task in different docker instance)
loop = asyncio.new_event_loop()
try:
asyncio.set_event_loop(loop)
loop.run_until_complete(asyncio.wait([_get_optimised_file(img, image_uri_map) for img in images_files]))
finally:
loop.close()
About _get_optimised_file() doesn't have any database operation, it checks if a file exists, if not, it sleeps for few seconds and check again.
Just try to understand any circumstance can cause a "async context" created and reuse by other WSGI request?
So far, I only can suspect 2 places. One is in settings.py, before I add both lines (I just disabled ASGI from settings in production env, need to observe if it does any fix)
WSGI_APPLICATION = 'myproj.wsgi.application'
ASGI_APPLICATION = 'myproj.asgi.application'
Second place is that asgi.py, if AuthMiddlewareStack breaks something in the sessions? But as my setup already separated ASGI and WSGI...
import os
from channels.security.websocket import AllowedHostsOriginValidator
from django.core.asgi import get_asgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproj.settings')
django_asgi_app = get_asgi_application()
from channels.auth import AuthMiddlewareStack
from channels.routing import ProtocolTypeRouter, URLRouter
import myproj.channel.routing
application = ProtocolTypeRouter({
'http': django_asgi_app,
"websocket": AllowedHostsOriginValidator(AuthMiddlewareStack(
URLRouter(
myproj.channel.routing.websocket_urlpatterns
)
)),
})