Why does my multiprocess queue not appear to be thread safe?

Viewed 418

I am building a watchdog timer that runs another Python program, and if it fails to find a check-in from any of the threads, shuts down the whole program. This is so it will, eventually, be able to take control of needed communication ports. The code for the timer is as follows:

from multiprocessing import Process, Queue
from time import sleep
from copy import deepcopy

PATH_TO_FILE = r'.\test_program.py'
WATCHDOG_TIMEOUT = 2

class Watchdog:

    def __init__(self, filepath, timeout):
        self.filepath = filepath
        self.timeout = timeout
        self.threadIdQ = Queue()
        self.knownThreads = {}

    def start(self):
        threadIdQ = self.threadIdQ

        process = Process(target = self._executeFile)
        process.start()
        try:
            while True:
                unaccountedThreads = deepcopy(self.knownThreads)

                # Empty queue since last wake. Add new thread IDs to knownThreads, and account for all known thread IDs
                # in queue
                while not threadIdQ.empty():
                    threadId = threadIdQ.get()
                    if threadId in self.knownThreads:
                        unaccountedThreads.pop(threadId, None)
                    else:
                        print('New threadId < {} > discovered'.format(threadId))
                        self.knownThreads[threadId] = False

                # If there is a known thread that is unaccounted for, then it has either hung or crashed.
                # Shut everything down.
                if len(unaccountedThreads) > 0:
                    print('The following threads are unaccounted for:\n')
                    for threadId in unaccountedThreads:
                        print(threadId)
                    print('\nShutting down!!!')
                    break
                else:
                    print('No unaccounted threads...')

                sleep(self.timeout)

        # Account for any exceptions thrown in the watchdog timer itself
        except:
            process.terminate()
            raise

        process.terminate()


    def _executeFile(self):
        with open(self.filepath, 'r') as f:
            exec(f.read(), {'wdQueue' : self.threadIdQ})

if __name__ == '__main__':
    wd = Watchdog(PATH_TO_FILE, WATCHDOG_TIMEOUT)
    wd.start()

I also have a small program to test the watchdog functionality

from time import sleep
from threading import Thread
from queue import SimpleQueue

Q_TO_Q_DELAY = 0.013

class QToQ:

    def __init__(self, processQueue, threadQueue):
        self.processQueue = processQueue
        self.threadQueue = threadQueue
        Thread(name='queueToQueue', target=self._run).start()

    def _run(self):
        pQ = self.processQueue
        tQ = self.threadQueue
        while True:
            while not tQ.empty():
                sleep(Q_TO_Q_DELAY)
                pQ.put(tQ.get())

def fastThread(q):
    while True:
        print('Fast thread, checking in!')
        q.put('fastID')
        sleep(0.5)

def slowThread(q):
    while True:
        print('Slow thread, checking in...')
        q.put('slowID')
        sleep(1.5)

def hangThread(q):
    print('Hanging thread, checked in')
    q.put('hangID')
    while True:
        pass

print('Hello! I am a program that spawns threads!\n\n')

threadQ = SimpleQueue()

Thread(name='fastThread', target=fastThread, args=(threadQ,)).start()
Thread(name='slowThread', target=slowThread, args=(threadQ,)).start()
Thread(name='hangThread', target=hangThread, args=(threadQ,)).start()

QToQ(wdQueue, threadQ)

As you can see, I need to have the threads put into a queue.Queue, while a separate object slowly feeds the output of the queue.Queue into the multiprocessing queue. If instead I have the threads put directly into the multiprocessing queue, or do not have the QToQ object sleep in between puts, the multiprocessing queue will lock up, and will appear to always be empty on the watchdog side.

Now, as the multiprocessing queue is supposed to be thread and process safe, I can only assume I have messed something up in the implementation. My solution seems to work, but also feels hacky enough that I feel I should fix it.

I am using Python 3.7.2, if it matters.

1 Answers

I suspect that test_program.py exits.

I changed the last few lines to this:

tq = threadQ
# tq = wdQueue    # option to send messages direct to WD

t1 = Thread(name='fastThread', target=fastThread, args=(tq,))
t2 = Thread(name='slowThread', target=slowThread, args=(tq,))
t3 = Thread(name='hangThread', target=hangThread, args=(tq,))

t1.start()
t2.start()
t3.start()
QToQ(wdQueue, threadQ)

print('Joining with threads...')
t1.join()
t2.join()
t3.join()

print('test_program exit')

The calls to join() means that the test program never exits all by itself since none of the threads ever exit.

So, as is, t3 hangs and the watchdog program detects this and detects the unaccounted for thread and stops the test program.

If t3 is removed from the above program, then the other two threads are well behaved and the watchdog program allows the test program to continue indefinitely.

Related