How to scrape a website's href (changing with session) from the inspect element, and not the source code of the page

user_14496135 0
Viewed 255

Requirement: [Trying to fetch the href from inspect element and store in a variable, then perform python requests.get() to fetch the data]
To get the COVID R0 data from the site : https://calcat.covid19.ca.gov/cacovidmodels/ into S3 , create a data pipeline and create a dashboard.

Steps to reproduce :

  1. Navigate to https://calcat.covid19.ca.gov/cacovidmodels/ -> Custom Datasets
  2. Select Current
  3. Select R-Effective
  4. By County -> Select All county from dropdown
  5. Right click on the Download Custom Dataset option :Copy link address
  6. Use this address to get the data [The link changes after every few minutes, and gets the latest session added to the URL]

Approach: I am able to get the dataset into S3, but since the link changes and contains a session, the script is unable to fetch the data.
When I do a view source code option, I do not see the href there, but "Inspect element" does show the latest href.
I am new to selenium, but whatever I could explore shows that it should be doable using selenium with python.

import requests 
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
options = webdriver.ChromeOptions()
options.add_argument('headless')
capa = DesiredCapabilities.CHROME
capa["pageLoadStrategy"] = "none"
driver = webdriver.Chrome(chrome_options=options, desired_capabilities=capa)
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver.set_window_size(1440,900)
driver.get('https://calcat.covid19.ca.gov/cacovidmodels/')
time.sleep(10)
xpath = "//*[contains(@href, ‘dl.canned?’)]"myElem = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH , xpath)))

Sample link : https://calcat.covid19.ca.gov/cacovidmodels/_w_3192292b/session/31bb334f1496250a1e17875f129054d7/download/dl.custom?w=3192292b

selenium-chromedriver python-requests web-scraping selenium beautifulsoup
1 Answers

You might want to try the head mode instead of the headless. For example this gets you Statewide R-Effective over Time canned dataset.

import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options


options = Options()
options.headless = False
driver = webdriver.Chrome(options=options)

driver.get("https://calcat.covid19.ca.gov/cacovidmodels/")
time.sleep(5)
driver.find_element_by_xpath('//*[@data-value="Downloads"]').click()
time.sleep(5)
driver.find_element_by_xpath('//*[@id="dl.canned"]').click()
time.sleep(3)
driver.close()

Then, check your default downloads folder.

EDIT:

To get the custom data set, I fooled around with selenium and came up with this. Hacky, but works. :-]

import time

from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys

options = Options()
options.headless = False
driver = webdriver.Chrome(options=options)

driver.get("https://calcat.covid19.ca.gov/cacovidmodels/")
time.sleep(5)
driver.find_element_by_xpath('//*[@data-value="Downloads"]').click()
time.sleep(5)
button = driver.find_element_by_xpath('//*[@data-id="cust.dl.bygeo"]')

(
    ActionChains(driver)
    .click(button)
    .send_keys(Keys.ARROW_DOWN)
    .send_keys(Keys.ENTER)
    .send_keys(Keys.ARROW_DOWN)
    .send_keys(Keys.ENTER)
    .send_keys(Keys.ESCAPE)
    .perform()
)

time.sleep(1)
driver.find_element_by_xpath('//*[@id="dl.custom"]').click()
time.sleep(10)
driver.close()
user_61067910
Related