To answer your questions, we need to understand some basics of threading and multiprocessing in python. It turns out that the issue is not so much the startup overhead, but rather how each distributes the running load over the system resources.
First, threading in python is not like threading in Linux. Where Linux creates a new lightweight process for each thread, and these can run on different CPU cores, the python script and its threads all run in the same CPU core at any given instant. If you want true multiprocessing in python, you have to use the multi processing interface.
To demonstrate the above, run the Linux system monitor, select the resources tab, and then in a different terminal window, try running each of the two code snippets that I inserted below. The resources tab shows you the load on each CPU core.
The second important issue, is that you want to handle thousands of incoming connections at the same time. You are probably going to need the multiprocessing interface for that, but there may be limits on how many processes and connections you can accommodate, either as configured in Linux or by bottlenecks in scheduling or resources, c.f. hardware.
One way to handle this, if you choose to not have a very large number of processes active at one time, is to create a fixed number of processes, store them in a list, and then pass the incoming connections to them as they come in. When all of the processes are busy, you wait. You will need at least one counting semaphore for this.
If you want to create processes as the connections come in, you can again use a counting semaphore to limit how many processes are running at one time. You initialize the counting semaphore to the maximum number, decrement it for each processes that you create, and increment it when the process exits. As above, when you reach the maximum number of allowed processes, you wait.
Okay, here are the code examples for threading and multiprocessing. The first starts 5 threads. The second starts 5 processes. You can change these with one edit, to work your way up to 100, 1000, etc. The integer processing loops in each, let you see the load in the Linux system monitor program.
#!/usr/bin/python
# Parallel code with shared variables, using threads
from threading import Lock, Thread
from time import sleep
# Variables to be shared across threads
counter = 0
run = True
lock = Lock()
# Function to be executed in parallel
def myfunc():
# Declare shared variables
global run
global counter
global lock
# Processing to be done until told to exit
while run:
n = 0
for i in range(10000):
n = n+i*i
print( n )
sleep( 1 )
# Increment the counter
lock.acquire()
counter = counter + 1
lock.release()
# Set the counter to show that we exited
lock.acquire()
counter = -1
lock.release()
print( 'thread exit' )
# ----------------------------
# Launch the parallel function in a set of threads
tlist = []
for n in range(5):
thread = Thread(target=myfunc)
thread.start()
tlist.append(thread)
# Read and print the counter
while counter < 5:
print( counter )
n = 0
for i in range(10000):
n = n+i*i
print( n )
#sleep( 1 )
# Change the counter
lock.acquire()
counter = 0
lock.release()
# Read and print the counter
while counter < 5:
print( counter )
n = 0
for i in range(10000):
n = n+i*i
print( n )
#sleep( 1 )
# Tell the thread to exit and wait for it to exit
run = False
for thread in tlist:
thread.join()
# Confirm that the thread set the counter on exit
print( counter )
And here's the multiprocessing version:
#!/usr/bin/python
from time import sleep
from multiprocessing import Process, Value, Lock
def myfunc(counter, lock, run):
while run.value:
sleep(1)
n=0
for i in range(10000):
n = n+i*i
print( n )
with lock:
counter.value += 1
print( "thread %d"%counter.value )
with lock:
counter.value = -1
print( "thread exit %d"%counter.value )
# -----------------------
counter = Value('i', 0)
run = Value('b', True)
lock = Lock()
plist = []
for n in range(5):
p = Process(target=myfunc, args=(counter, lock, run))
p.start()
plist.append(p)
while counter.value < 5:
print( "main %d"%counter.value )
n=0
for i in range(10000):
n = n+i*i
print( n )
sleep(1)
with lock:
counter.value = 0
while counter.value < 5:
print( "main %d"%counter.value )
sleep(1)
run.value = False
for p in plist:
p.join()
print( "main exit %d"%counter.value)
