I have some code that will download some json from an api. I have to go through many pages and requests.get is very slow so I'm trying to use multiprocessing to speed things up. Here is my code:
def worker(mod, offset, totalpages, arr):
stuff = []
for i in range(offset, totalpages, mod):
data = requests.get("https://api.website.net?" + "page=" + str(i)).json()
try:
stuff += data["list"]
except KeyError:
pass
print(str(i + 1) + " pages downloaded")
for x in stuff:
arr.append(x)
def getdata(threads):
totalPages = requests.get("https://api.website.net?page=0").json()["totalPages"]
proccesses = []
items = []
for i in range(threads):
p = multiprocessing.Process(target=worker, args=(threads, i, totalPages, items))
p.start()
proccesses.append(p)
for p in proccesses:
p.join()
print(items)
getdata(4)
This ends up printing an empty array, even though I'm using append which should change the referenced list. Is there any way I can change my code so that worker modifies the array that is passed to it instead of doing nothing?
edit: I figured it out, I have to use multiprocessing.Manager() to create a list that all the threads can access.