how to use threading.Semaphore with ThreadPoolExecutor in python

user_11911577 0
Viewed 300

I want to know how I can use something like Semaphore in ThreadPoolExecutor My code is like this

def download_one(url):
    requests.get(url)
    ...
    ...
    ...
    return 'ok'

def download_all(urls):
    with ThreadPoolExecutor(max_workers=7) as executor:
        return executor.map(download_one, urls, timeout=20)

download_all(["url-1", "url-2"])

My code works very well this way But the problem I have is that I may call the download_all function several times with different urls An update comes to my program and those updates are the urls and I want to transfer them to the download_all function.

However, I want it to work when I first call the download_all function And when I call it a second time, wait until it's done and then start again

So that there are always 7 workers running and not added to the workers when the function is added several times.

with ThreadPoolExecutor(max_workers=7) as executor:

concurrent.futures threadpoolexecutor concurrency multithreading python
1 Answers

If I'm understanding your question correctly, simply make sure that you are using only a single executor for all of your calls. Something like the following should work:

def download_one(url):
    requests.get(url)
    ...
    ...
    ...
    return 'ok'

def download_all(executor, urls):
    return executor.map(download_one, urls, timeout=20)

with ThreadPoolExecutor(max_workers=7) as executor:
    ...
    download_all(executor, ["url-1", "url-2"])
    ...
    download_all(executor, ["url-3", "url-4", "url-5"])
    ...
    download_all(executor, ["url-6"])
user_50612770
Related