I am trying to scrape a traffic camera website. I recently asked a question on how to do it for a map of Houston, but I now would like to do it to two other maps in NYC and Boston. In the previous case, you could find the url to GET request from for a data frame that contained information on the camera data by going to inspect element -> networks -> and looking at links that were get requested from. I am trying to do the same thing here, but when I try to do a get request from these URLs I'm usually met with an error.
This is the nyc website: https://webcams.nyctmc.org/ This is the boston website: https://www.mass.gov/info-details/massachusetts-traffic-map
I'll take the nyc website as an example. An error I get with trying to do a get request with one of the URLs in the networks section to get a potential data frame is:requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
You can get an example traffic camera feed by clicking on one of the blue camera icons on the map and you get this popup.
By inspect element of that websites URL, I can see that the image has a source: https://jpg.nyctmc.org/cctv474.jpg?math=0.3197823215689304
However if i try to Get request this source link I get this error: requests.exceptions.SSLError: HTTPSConnectionPool(host='jpg.nyctmc.org', port=443): Max retries exceeded with url: /cctv474.jpg?math=0.3197823215689304 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)'))) (I think because the source link keeps changing every few seconds).
I need help with:
- Finding a data frame for the locations / overall data about the all the cameras on the traffic map
- Using that data to request the images at each camera feed
I am pretty new to this space so any kind of resource / education on how to approach finding data frames in websites would be useful as well!