multiprocessing.Queue get() works from a Process but not another Thread

Viewed 114

I feel like I am losing my mind here.

My program creates a Multiprocessing Queue, then spawns a new Process. From within that Process, I can call get() on the queue and receive objects. However, if I spawn a new Thread from within that Process, the Thread CANNOT receive anything when calling get(). I also tried it with a Manager.Queue and got the same result.

Here is a simplified snippet.

    from multiprocessing import Process, Queue
    import threading

    def create_process():
        """ Create the queue and start the process """
        cmd_queue = Queue()

        my_proc = Process(target=process_run, args=(cmd_queue,), daemon=True)
        my_proc.start()

        # DEBUGGING:
        time.sleep(1)
        cmd_queue.put("test1")
        time.sleep(1)
        cmd_queue.put("test2")

    def process_run(cmd_queue:Queue):
        """ Function to be run as a multiprocessing Process."""
        logger.info("process_run get(): %s", cmd_queue.get())
    
        # Set up thread to receive commands:
        thd = threading.Thread(target=thread_commands, args=(cmd_queue,), name="Commands", daemon=True)
        thd.start()

    def thread_commands(cmd_queue:Queue):
        """ Thread to retrieve commands from the command queue and process them """
        while True:
            logger.info("thread_commands get(): %s", cmd_queue.get())

When I run this, I get the message "process_run get(): test1", but "thread_commands" never returns from get(). What am I doing wrong here?

1 Answers

Yep it was me being dumb.

I started the Thread with daemon=True and then the Process exited, thus killing the thread.

Related