Use objects in parallel processing and "restart" these

user_7885426 0
Viewed 53

The goal

I want to retrieve information from webpages (in parallel). As soon as one of the "crawlers" finds the result we are looking for we terminate, if not we refresh the page we just looked at and search again. To put this differently:

  • Open the webpages in 3 processes (same page, delayed X sec)
  • Return the result as soon as we have it (per process not all at once)
  • If this result ==True we are done and terminate the pool
  • if not, we call .restart() and add it to the pool again
  • repeat

The code side

Scrape class

Lets first define the Scraper object:

import random
import time
import multiprocessing

# Result simulation array, False is much more likely than True
RES = [True, False, False, False, False, False, False, False, False, False, False, False, False]

class Scrape:

    def __init__(self, url):
        self.url = url
        self.result = None

    def get_result(self):
        return self.result

    def scrape(self):
        # Go to url
        # simulate work
        time.sleep(random.randrange(5))
        # simulate result
        result = RES[random.randrange(13)]
        # Return what we found on the page
        self.result = result

    def restart(self):
        # >> Some page refreshing
        self.scrape()

So we go to the webpage and do some work (scrape) then we can access the results via get_result and if this is not what we want we can call restart. Note that in reality this class is much more complex so creating it all over again would be a waste of starting up the driver (compared to reusing the same class via restart)

Parallel code

Here is where I got stuck, while I used map hundreds of times I don't know how I can keep the Scrape objects and call restart and add them to the pool again. I was thinking of something like this, but this does not work as I want it to. Perhaps a queue is a better approach for this but I'm not familiar with it.

# Function to create the scrapers
def obj_create(url):
    print('Create')
    a = Scrape(url)
    a.scrape()
    return a

# Function to restart the scraper
def obj_restart(a):
    print('Restart')
    a.restart()
    a.scrape()
    return a

# Callback
def call_back(scrape_obj):
    if scrape_obj.get_result():
        pool.terminate()
        # Also somehow return the result...
    else:
        # Restart and add again
        pool.apply_async(obj_restart, scrape_obj, callback=call_back)


pool = multiprocessing.Pool(3)
url = 'test.com'
delay = 0.001
n = 3

for i in range(n):
    time.sleep(delay)
    pool.apply_async(obj_create, url, callback=call_back)

pool.close()
pool.join()

I tried my best to make this reproducible example, and explain it as well as I could, but please let me know if anything is unclear!

parallel-processing asynchronous python performance
1 Answers

python

in parallel

Seems you never heard about the GIL yet? If so, I am sorry to disappoint you, but you can't get true parallelism because of it.

More than that. In the case you've described.. you don't ever need one. Because most of the time your threads will be blocked by the network interface to reply back. Thus what you need is a good async framework, a-ka event loop, that will optimize your execution such that blocking reduces to a possible minimum.

Let me introduce aiohttp. It gives you rightly implemented HTTP(s) calls and the glorious asyncio provides you with neat mechanism to cancel, retry and handle errors.

In particular, have a look at the gather.

P.S. multiprocessing will make things parallel, yet it is an overkill here due to IO-bound nature of the task. You may stay within the same process and avoid all the multiprocessing hell simply by rejecting this kind of parallelism which either way will be blocked most of the time.

user_74938400
Related