Implementing thread pool for large number of thread classes

user_654789 0
Viewed 104

There are many examples online for thread pools but almost all deal with the concept of passing functions around and calling those (like here or here). But I have some objects which inherit from threading.Thread and I want only some of them to be running at any time. Here is a minimal working example

class Human (threading.Thread):
    def run (self):
        print ("taking a nap")
        sleep (60*60*8)

human_population = 7000000000

for i in range(human_population):
    human=Human()
    human.start()

Ignoring for the mmoment that 7B objects would crush my computer, I am looking for a very easy way to run only a manageable number of threads at any time, for example, N= 10 threads in a FIFO fashion.

threadpool python-3.x multithreading python
1 Answers

Semaphore is a very fitting tool for these cases. You can use as semaphore with initial size of 10 to control how many threads can be active at most at a time.

A rough implementation over yours (should probably not use a global variable for the semaphore, and handle lifecycles more cleanly):

from threading import Thread, Semaphore
from time import sleep

SEM = Semaphore(10)
 
class Human(Thread):
    def run(self):
        print("taking a nap")
        sleep(5)
        SEM.release() # increments semaphore counter, notifying the job's done

human_population = 20

for i in range(human_population):
    SEM.acquire() # this will block when semaphore has value 0, and will wait until one of the active one finishes and calls release()
    human = Human()
    human.start()
user_69166310
Related