Keep Selenium Webdriver across Celery Tasks

Question

I developed a Flask web app that uses Celery to handle tasks. One of these tasks consists of scraping a bunch of pages (around 200) using a custom Class derived from a selenium chrome driver.

@celery_app.task
def scrape_async():
   driver = MyDriver(exec_path=os.environ.get('CHROMEDRIVER_PATH'), options=some_chrome_options)
   
   # Update 'urls_to_scrape' attribute by finding the urls to scrape from a main url
   driver.find_urls_to_scrape_from_main_url()
   
   # Loop over each page and store the data in database
   for url in driver.urls_to_scrape:
      driver.scrape_page(url)

  # Exit driver
  driver.quit()

This worked well both in local and in production until the number of pages to scrape increased. I received a memory usage error on Heroku and realized how memory intensive the task was.

After some research, I found out that it would be easier to use a subtask in my loop (i.e. running this subtask for each page), as described below.

@celery_app.task
def subtask(url):
  driver.scrape_page(url)

@celery_app.task
def scrape_async():
   driver = MyDriver(exec_path=os.environ.get('CHROMEDRIVER_PATH'), options=some_chrome_options)
   
   # Update 'urls_to_scrape' attribute by finding the urls to scrape from a main url
   driver.find_urls_to_scrape_from_main_url()
   
   # Loop over each page and store the data in database
   for url in driver.urls_to_scrape:
      subtask.delay(url)

  # Exit driver
  driver.quit()

My concern is about how to keep the driver object between the main task and the subtask. I found this link, on how to instantiate Task but I couldn't manage to create one driver across multiple tasks.

Any idea on how to proceed to achieve my goal?

Thank you

Keep Selenium Webdriver across Celery Tasks

0 Answers