Real-time scraping website with Django: stoppable scraping process and loading bar

user_9093280 0
Viewed 288

What I would like to achieve

I would like to make a website that allows users to scrape information from other websites. My goal is to achieve something as follows:

  1. User presses "scrape"
  2. When the scraping process begins, Django renders a view with a loading message and a "stop" button
  3. If the user presses the "stop" button, or if the scraping process terminates, a view with the scraping results is shown

Current state

At the moment, I have a very simple scraping website with a horrible user experience. After the user presses scrape, I render a view similar to the following one:

def scraping_view(request, scraper_args):
    scrape_results = scrape(scraper_args) # Very long function!
    return render(request, 'scrape_view.html', scrape_results )

This is a bad idea, as the scrape function takes a long time, and the user experience is terrible.

My (probably bad) idea

My idea was to:

  • launch a celery task and render an intermediary view with the loading message and a "stop" button
  • let the celery task store the scrape results somewhere (where?)
  • find a way to revoke the task if the user presses the "stop" button
  • retrieve the results of the celery task when the task is finished or when "stop" is being pressed
  • cancel the scrape results after the page is closed, as I am simply interested in showing them on the website

As I am new to Django, I cannot evaluate if my idea is feasible or if it makes sense at all. Does anyone have a solution to propose, an alternative solution, or some examples to share? Thanks!

celery web-scraping django python
1 Answers

You are on the right track with using celery, so celery will manage the queue of scraping tasks and execute them with a pool of workers, celery allows for you to store custom processing states see here: https://docs.celeryproject.org/en/stable/userguide/tasks.html#custom-states

You will have and endpoint that will trigger a celery task execution to begin scraping. When a selenium scraping tasks begins celery will return a task ID which you can store and use for later reference.

Celery also allows the tasks to be revoked/cancelled by the celery task id see here: https://docs.celeryproject.org/en/latest/userguide/workers.html#revoke-revoking-tasks , this will be your cancel button functionality.

In terms of updating the user via the website of the scraping progress, I would use celery in combination with django-channels and establish a web socket connection on your front end which will keep the scraping progress real time and avoid need for polling some rest endpoint: https://channels.readthedocs.io/en/stable/

user_34761340
Related