Python - multiprocessing reading from an array and updating a tally

cuppatea3 0
Viewed 28

I have some tests that run in locust and as part of it, I create a unique ID and add it to an array. I would then like to run through those ID's at the end and check if they made it into the DB via a web call. It works perfectly fine but it is quite slow to run through each individually... so I believe I need to use multiprocessing (or maybe some other method?) to speed up the check, but so far I have not managed to get it to work.

Here is an example of code that I have written (obfuscated by sausages because... why not :) )

In this example the array only has 3 items in it, but if you can imagine that it has hundreds, what is the best way to make this multiprocessing please?

import requests

all_sausages = ["cumberland", "lego", "bratwurst"]
global yes 
global no 
yes = 0
no = 0
for sausage in all_sausages:
    response = requests.get("http://www.isitasausage.com/" + sausage)
    if response.text.__contains__("Yes, this is a sausage"):
        yes += 1 
    else:
        no =+ 1 

print("Number of sausages in the array = " + str(yes))
print("Number of items in the array that are not sausages = " + str(no))
locust python
2 Answers

Like this, for a tiny example.

I'm using ThreadPool instead of a process pool because

  • dealing with HTTP is mostly IO-bound, not CPU-bound so the GIL doesn't matter
  • you can share a single requests.Session() like this
  • you don't need to pay the serialization tax associated with multiprocessing.
import requests
from collections import Counter
from multiprocessing.dummy import ThreadPool


sess = requests.Session()


def determine_sausageness(sausage: str) -> tuple[str, bool]:
    # Return the sausageness of a given sausage; returns a tuple of (sausage, result).
    response = requests.get("http://www.isitasausage.com/" + sausage)
    response.raise_for_status()
    return (sausage, "Yes, this is a sausage" in response.text)


def main():
    all_sausages = ["cumberland", "lego", "bratwurst"]

    with ThreadPool() as pool:
        results = dict(pool.imap_unordered(determine_sausageness, all_sausages))
    responses = Counter(results.values())
    print(responses)


if __name__ == "__main__":
    main()
AKX0

Thanks so much for the replies, they were very useful. Also, sorry for the typo in my question, I knocked it up quickly and missed that :)

The solution I went with in the end was to use ThreadPool as suggested and this worked a treat for me. I did a 4 min test initially and it took nearly 15 mins to process the 1500 requests one at a time once the 4 min test had completed, but with the following code it reduced from 15 mins to 48 seconds! Quite the improvement! (NB I also edited the array to contain the URL, but this is not necessarily needed)

import concurrent.futures
import requests

[all_sausages] = ["http://www.isitasausage.com/cumberland",
"http://www.isitasausage.com/generic_toy_blocks",
"http://www.isitasausage.com/bratwurst",
"http://www.isitasausage.com/hotdog",
"http://www.isitasausage.com/dachshund",
"http://www.isitasausage.com/chorizo"]

def determine_sausageness(sausage):
    global yes
    global no
    response = requests.get(sausage)
    if response.text.__contains__("Yes, this is a sausage"):
        yes += 1
    else:
        no += 1

with concurrent.futures.ThreadPoolExecutor() as executor:
    executor.map(determine_sausageness, all_sausages)

print("Number of sausages in the array = " + str(yes))
print("Number of items in the array that are not sausages = " + str(no))
cuppatea30
Related