BeautifulSoup Detect Change Trigger

user_7221994 0
Viewed 1953

I have a script that uses bs4 to scrape a webpage and grab a string named, "Last Updated 4/3/2020, 8:28 p.m.". I then assign this string to a variable and send it in an email. The script is scheduled to run once a day. However, the date and time on the website change every other day. So, instead of emailing every time I run the script I'd like to set up a trigger so that it sends only when the date is different. How do I configure the script to detect that change?

String in HTML:

COVID-19 News Updates




Last Updated 4/3/2020, 12:08 p.m.

'''Checks municipal websites for changes in meal assistance during C19 pandemic'''

# Import requests (to download the page)
import requests
# Import BeautifulSoup (to parse what we download)
from bs4 import BeautifulSoup
import re

#list of urls
urls = ['http://www.vofil.com/covid19_updates']

#set the headers as a browser
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

#download the homepage
response = requests.get(urls[0], headers=headers)
#parse the downloaded homepage and grab all text
soup = BeautifulSoup(response.text, "lxml")
#Find string
last_update_fra = soup.findAll(string=re.compile("Last Updated"))
print(last_update_fra)

#Put something here (if else..?) to trigger an email.

#I left off email block...

The closet answer I found was this, but it's referring to the tags, not a string. Parsing changing tags BeautifulSoup

beautifulsoup python-3.x python
1 Answers

You would need to check every so often for a change using something along the lines of this:

'''Checks municipal websites for changes in meal assistance during C19 pandemic'''

# Import requests (to download the page)
import requests
# Import BeautifulSoup (to parse what we download)
from bs4 import BeautifulSoup
import re
import time

#list of urls
urls = ['http://www.vofil.com/covid19_updates']

#set the headers as a browser
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

while True:
    #download the homepage
    response = requests.get(urls[0], headers=headers)
    #parse the downloaded homepage and grab all text
    soup = BeautifulSoup(response.text, "lxml")
    #Find string
    last_update_fra = soup.findAll(string=re.compile("Last Updated"))
    time.sleep(60)
    #download request again
    soup = BeautifulSoup(requests.get(urls[0], headers=headers), "lxml")

    if soup.findAll(string=re.compile("Last Updated")) == last_update_fra:
        continue
    else:
        # Code to send email
*This waits a minute before checking for changes, but it can be adjusted as needed.
[Credit][1]: https://chrisalbon.com/python/web_scraping/monitor_a_website/
user_120278220
Related