How to make a continuous parser from a regular one?

Question

I have a parser that saves data to a json file. Each time it starts, it parses the specified information and exits.

How to make it work continuously? So that I do not run it all the time, but it works continuously, and as soon as someone adds a product, he will immediately add a new topic to the base.

Here's a parser and an example of the base that it parses :

main.py

import json
import requests
from bs4 import BeautifulSoup


def get_first_news():
    headers = {
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.101 Safari/537.36"
    }

    url = "https://funpay.ru/lots/700/"
    r = requests.get(url=url, headers=headers)

    soup = BeautifulSoup(r.text, "lxml")
    articles_cards = soup.find_all(
        "a", class_="tc-item")

    news_dict = {}
    for article in articles_cards:
        article_title = article.find("div", class_="tc-desc-text").text.strip()
        article_desc = article.find("div", class_="tc-price").text.strip()
        article_url = f'{article.get("href")}'

        article_id = article_url.split("=")[-1]

        # print(f"{article_title} | {article_url} | {article_date_timestamp}")

        news_dict[article_id] = {
            "article_title": article_title,
            "article_url": article_url,
            "article_desc": article_desc
        }

    with open("news_dict.json", "w", encoding='utf-8') as file:
        json.dump(news_dict, file, indent=4, ensure_ascii=False)


def main():
    get_first_news()


if __name__ == '__main__':
    main()

news_dict.py

{
    "7705701": {
        "article_title": "ACCOUNT 830 SINGLE STRIKE SUBSCRIBERS, Video games, 830 subscribers",
        "article_url": "https://funpay.ru/lots/offer?id=7705701",
        "article_desc": "742.86 ₽"
    },
    "7691069": {
        "article_title": "⭐⚡2 CHANNELS FOR ONE PURCHASE! | CHEAP TOGETHER! | MEGA PACK CHANNELS! ⭐⚡, Video Games, 999999 subscribers",
        "article_url": "https://funpay.ru/lots/offer?id=7691069",
        "article_desc": "7428.57 ₽"
    },
    "7653340": {
        "article_title": "TOP MIKNCRAFT CHANNEL, ADVISE EVERYONE! [GOOD BARGAINING POSSIBLE] ❤️✨, Video Games, 7300 subscribers",
        "article_url": "https://funpay.ru/lots/offer?id=7653340",
        "article_desc": "5200.00 ₽"
    },
    "7582336": {
        "article_title": "⚡⚔️CHANNEL WITH A VERY GOOD ASSET: AUDIENCE - KAZAKHI (ADULTS OVER 18 YEARS OLD) ⚡⚔️, News and politics, 2800 subscribers",
        "article_url": "https://funpay.ru/lots/offer?id=7582336"
    }

user_3804427 · Answer

Usualy we use system task manager (cron, microsoft task manager) for start script every minute or every 10 minute. You can use endless circle too (while True)!

import time
def main():
    while True:
        get_first_news()
        time.sleep(1)

How to make a continuous parser from a regular one?

1 Answers