Threadpool and dynamicly changing number of workers

user_14309671 0
Viewed 608

Lets say I want to to implement this code but the MAIN a differences are :

  1. I dont know what is the max_workers value.
  2. My workers must work in parallel.
  3. My number of workers are changing dynamicly.

I thought about making a queue of workers and by that keep pushing tasks to the available worker but I dont know how to do / implement it. Thank you very much. I hope my question is clear.

import concurrent.futures
import urllib.request

URLS = ['http://www.foxnews.com/',
   'http://www.cnn.com/',
   'http://europe.wsj.com/',
   'http://www.bbc.co.uk/',
   'http://some-made-up-domain.com/']

def load_url(url, timeout):
   with urllib.request.urlopen(url, timeout = timeout) as conn:
   return conn.read()

with concurrent.futures.ThreadPoolExecutor(max_workers = 5) as executor:

   future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
   for future in concurrent.futures.as_completed(future_to_url):
   url = future_to_url[future]
   try:
      data = future.result()
   except Exception as exc:
      print('%r generated an exception: %s' % (url, exc))
   else:
      print('%r page is %d bytes' % (url, len(data)))
threadpoolexecutor threadpool python
1 Answers

You can take the number or urls divided by 10 as the max workers identifier, since their count impacts the count of jobs you are going to submit:

max_workers = URLS//10
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
    ...
user_20679760
Related