Multiprocessing in python vs number of cores

user_13569767 0
Viewed 2135

If a run a python script where i declare 6 processes using multiprocessing, but i only have 4 CPU cores, what happens to the additional 2 processes which can find a dedicated CPU core.

  1. How are they executed?
  2. If the two additional processes run as separate threads on the existing Cores, will GIL not stop their execution?

#Edit 1 - 21st Jan 2021

I have mixed up threads and processes in the question I asked. Since I have better clarity on the concept, I would rephrase question 2 as follows(for any future reference):

If the two additional processes run in parallel with two other processes in existing Cores, will GIL not stop their execution?

Ans: GIL does NOT affect the processes, GIL allows only one thread to run at a time, there is no restriction on processes, however. The system scheduler manages how the additional two processes will run on the existing cores.

cpu-cores multiprocessing python
2 Answers

First you are mixing up threads and processes: in Python only threads not processes have to share a lock on their interpreter. If your are using the multiprocessing library then, your are using Python processes which have their own interpreter.

When you are using Python processes, their execution is managed by your operating system scheduler, in the same manner as every other processes in your computer. If you have more processes than CPU cores then the extra processes are waiting in background to be scheduled. This usually happen when an other process terminates, wait on an IO, or periodically with clock interrupts.

user_140480130

It is always best practice to make sure that you use

pool = multiprocessing.Semaphore(multiprocessing.cpu_count() - 1) 
#this will detect the number of cores in your system and creates a semaphore with that value.

When you create a process it takes overhead to manage it, its memory space, and its shared memory. Also, the operating system has to run, so leaving a core free is always polite and speeds up the execution of the problem.

user_54783730
Related