Made a web scraping program in python, it was slow so i used some multithreading, it worked well so I tried multiprocessing, used basically the same function just swapping out the bits relevant for the different packages.
t = []
j=2
for _ in range(cores):
process = threading.Thread(target=do_job, args=(j,num,cores,))
process.start()
t.append(process)
j=j+1
for x in t:
x.join()
^ this is the multithreading code, it works as expected, it wrote to the file like I wanted.
t = []
j=2
for _ in range(cores):
process = Process(target=do_job, args=(j,num,cores,))
process.start()
t.append(process)
j=j+1
for x in t:
x.join()
^ this is the multiprocessing code, it doesn't work, no errors and nothing written to the file. Am I missing something obvious? It shouldn't be because of the rest of my code but I will still post the function these call.
def do_job(ini,num,cores):
for i in range(ini,num,cores):
url = 'https://www.imovirtual.com/comprar/apartamento/?page=' + str(i)
page = requests.get(url)
print(page)
soup = BeautifulSoup(page.content, 'html.parser')
lists = soup.find_all('div', class_="offer-item-details")
get_info(lists,i)