Multithreading using queues in django + uWSGI

Question

I am struggling to make Python standard queues work in uWSGI master.

Sample code:

import queue, threading, time
from django.apps import AppConfig

my_queue = queue.Queue()

class MyApp(AppConfig):
    def ready(self):
        thread1 = threading.Thread(target=process_queue, args=())
        thread1.daemon = True
        thread1.start()

        thread2 = threading.Thread(target=poll, args=())
        thread2.daemon = True
        thread2.start()

def poll():
    while True:
        time.sleep(1)
        logging.info("Polling works every time")


def process_queue():
    while True:
        logging.info("I get here")
        email = my_queue.get(block=True, timeout=None)
        logging.info("I NEVER GET here. **I would like to get here**")

def put():
    # I call this from the django views on requests
    my_queue.put("1")

Both threads start. Thread 2 prints at regular intervals as expected. Thread 1 waits on the queue forever, even though I add to it from main thread (As part of Django requests, inside the views)

My uWSGI config:

[uwsgi]
master         = true
module         = backend.wsgi
processes      = 1
thunder-lock   = true
enable-threads = true
socket         = /tmp/django.sock
chmod-socket   = 666
vacuum         = true

Notice that if I run uWSGI with master = false, everything works as expected

Additional information:

I am running this as a vassal. The emperor is not running as master
I do not really understand pre/post forking in django + uWSGI. This could be an issue
This works if not ran as master, but I cannot afford doing that

Versions:

Python: 3.6.3
Django: 2.0.5
uWSGI: 2.0.17

I opened this issue as well https://github.com/unbit/uwsgi/issues/1860

Multithreading using queues in django + uWSGI

0 Answers