Add web scraping data into the database at regular intervals [ruby & ror]

Viewed 32

I'm learning full stack development through the free course at appacademy. If you are familiar with it. am almost finished with the rails section and have done also done the SQL & Ruby sections. I have yet to start JavaScript or React.

To test my abilities, I want to create an app. It will mostly be backend + HTML since I haven't covered front end yet.

The app function: ability for someone to provide a list of URLs for reddit posts and then track number of upvotes. I want to do this by scraping the reddit content using this. This is just to test my ability and not a real use case.

I only want to scrape once a day to keep the scraping function minimal. I also want to add a refresh button next to each post, so the user can refresh when they want to.

Questions:

Ill be creating a database that stores the value of upvotes. However, from what I have learned with the program, I do not know how I can use my ruby app to keep running my daily scraper function. If I create a file scraper.rb with the function that is on a time loop, how do I put this file in my ror created app? do i put it in the models, views or controllers folder? Will it run automatically if I run my ruby server? or is this part entirely separate? do I run two ruby apps at once? One for backends and one the actual page

For the refresh button, I think I would need to include a call to the scraper function in my controller before rendering the refreshed text. Does that sound right?

1 Answers

From your description, I would say that the scraper should be a model method and it needs to be wrapped around a rake task.

There are many ways you can achieve the periodic execution. Some of them are:

  1. Whenever gem
  2. Plain old cron job for running the rake task
  3. If you deploy it in a PaaS like heroku, render.com or something similar, there are addons which can be configured to do the cron,like heroku scheduler or render.com cron.
Related