Multithreading working but not multiprocessing in python

Viewed 25

Made a web scraping program in python, it was slow so i used some multithreading, it worked well so I tried multiprocessing, used basically the same function just swapping out the bits relevant for the different packages.

t = []
j=2
for _ in range(cores):
    process = threading.Thread(target=do_job, args=(j,num,cores,))
    process.start()
    t.append(process)
    j=j+1
for x in t:
    x.join()

^ this is the multithreading code, it works as expected, it wrote to the file like I wanted.

t = []
j=2
for _ in range(cores):
    process = Process(target=do_job, args=(j,num,cores,))
    process.start()
    t.append(process)
    j=j+1
for x in t:
    x.join()

^ this is the multiprocessing code, it doesn't work, no errors and nothing written to the file. Am I missing something obvious? It shouldn't be because of the rest of my code but I will still post the function these call.

def do_job(ini,num,cores):

    for i in range(ini,num,cores):
        url = 'https://www.imovirtual.com/comprar/apartamento/?page=' + str(i)
        page = requests.get(url)
        print(page)
        soup = BeautifulSoup(page.content, 'html.parser')
        lists = soup.find_all('div', class_="offer-item-details")
        get_info(lists,i)
0 Answers
Related