Wrong threading.active_count() results when using ThreadPool

Question

I am using the multiprocessing.pool.ThreadPool with N threads (e.g 5 threads) and I wanted to check the total number of active threads in my process. To do that I am using the method threading.active_count(). I know it's a different module, but I found no other method to count the number of active threads in the multiprocessing package,

The expected result is N+1 (the number of threads I started plus the main thread), but I always get a higher number.

For ThreadPool(2) I am getting 6 active threads

For ThreadPool(5) I am getting 9 active threads

For ThreadPool(10) I am getting 14 active threads

It's important to say that threading.active_count() works fine when creating threads using the threading module. And I found out that multiprocessing.pool.ThreadPool is not well documented.

Can someone help me?

A reproduceable code is described bellow

import threading
from multiprocessing.pool import ThreadPool
import time
import requests
import os

urls_to_download = [
    'https://picsum.photos/seed/1/1920/1080',
    'https://picsum.photos/seed/2/1920/1080',
    'https://picsum.photos/seed/3/1920/1080',
    'https://picsum.photos/seed/4/1920/1080',
    'https://picsum.photos/seed/5/1920/1080',
    'https://picsum.photos/seed/6/1920/1080',
    'https://picsum.photos/seed/7/1920/1080',
    'https://picsum.photos/seed/8/1920/1080',
    'https://picsum.photos/seed/9/1920/1080',
    'https://picsum.photos/seed/10/1920/1080',
    'https://picsum.photos/seed/11/1920/1080',
    'https://picsum.photos/seed/12/1920/1080',
    'https://picsum.photos/seed/13/1920/1080',
    'https://picsum.photos/seed/14/1920/1080',
    'https://picsum.photos/seed/15/1920/1080',
    'https://picsum.photos/seed/16/1920/1080',
    'https://picsum.photos/seed/17/1920/1080'
]



output_dir = 'downloaded_images'

## 

def download(url):
    print(f'downloading {url}')
    img_data = requests.get(url).content
    img_name = url.split('/')[-3]
    img_name = f'{img_name}.jpg'
    print(f'Received data for {img_name}')
                
    print(f'Active Threads: {threading.active_count()}')
    
    with open(os.path.join(output_dir,img_name), 'wb') as img_file:
        img_file.write(img_data)
        
        
number_of_threads = 2
     
t1 = time.perf_counter()

with ThreadPool(number_of_threads) as pool:
    pool.map(download,urls_to_download)
    
t2 = time.perf_counter()
print(f'Finished in {t2-t1} seconds')

Wrong threading.active_count() results when using ThreadPool

0 Answers