How do I make a process modify a list passed to it in python?

Viewed 42

I have some code that will download some json from an api. I have to go through many pages and requests.get is very slow so I'm trying to use multiprocessing to speed things up. Here is my code:

def worker(mod, offset, totalpages, arr):
    stuff = []
    for i in range(offset, totalpages, mod):
        data = requests.get("https://api.website.net?" + "page=" + str(i)).json()
        try:
            stuff += data["list"]
        except KeyError:
            pass
        print(str(i + 1) + " pages downloaded")
    for x in stuff:
        arr.append(x)
def getdata(threads):
    totalPages = requests.get("https://api.website.net?page=0").json()["totalPages"]
    proccesses = []
    items = []
    for i in range(threads):
        p = multiprocessing.Process(target=worker, args=(threads, i, totalPages, items))
        p.start()
        proccesses.append(p)
    for p in proccesses:
        p.join()
    print(items)
getdata(4)

This ends up printing an empty array, even though I'm using append which should change the referenced list. Is there any way I can change my code so that worker modifies the array that is passed to it instead of doing nothing?

edit: I figured it out, I have to use multiprocessing.Manager() to create a list that all the threads can access.

1 Answers

use multiprocessing.Manager() to create a list that all threads can access

Related