FastAPI - Best way to run continuous GET requests in the background

user_15091041 0
Viewed 2068

I am trying to create a program which does periodic GET requests from around 10 websites and updates the information in a DB locally. Now when a user wants information, I will display the locally stored aggregate info.

I am trying to figure out the best way to run these periodic GET requests in FastAPI. I am new to FastAPI and am still trying to figure things out.

After some research I can think of two options:

  1. Using a background task endpoint which runs periodically and does the GET requests one by one from each website.
  2. Using Celery to do these GET requests

If anyone has any experience of doing something similar I am trying to figure out the best way to do this, or how would I go about finding out the best way to do this?

fastapi celery python
1 Answers

I think that for that Celery is overkill for one periodic task. Celery requires a broker (and preferable backend as well). Are you going to bringup/manage RabbitMQ only for a single task?

A simpler solution can be using asyncio for that:

TIME_INTERVAL_IN_SEC = 60


async def crawl_websites():
    while True:
        # async GET requests
        # async update DB
        await asyncio.sleep(TIME_INTERVAL_IN_SEC)


loop = asyncio.get_event_loop()
task = loop.create_task(crawl_websites())
loop.run_until_complete(task)
user_10112530
Related