I'm trying to send a notification every time a new post is made on a given craigslist page. Here's an excerpt of my code:
import requests, logging, http.client
http.client.HTTPConnection.debuglevel = 1
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
url = "https://losangeles.craigslist.org/search/cta?query=honda+civic"
page = requests.get(url)
If you run this code, you will see this line in the output:
header: Date: Wed, 13 Jul 2022 19:31:21 GMT (obviously with the date and time that you run it).
The problem is, if you run the code again, this line doesn't update. You can send as many requests as you want, but it will keep giving you the same stale page for the next 15 minutes. (Notice this line as well: Cache-Control: max-age=900 15 minutes=900 seconds) I've tried adding different headers ('user-agent':'Chrome etc', 'Cache-Control':'no-cache', etc) but they do nothing. I even tried cycling through a list of proxies to send the requests but I still get stale pages.
When I run the code with the URL set to my own website, it updates on each request. No issues.
Can any web experts out there help me?