Issue Scraping data from website that seems to change

user_10604556 0
Viewed 61

I'm attempting to scrape the data from the PR College table on https://gbr.milesplit.com/athletes/pro/5424599/stats The Code looks like

prTable = wait.until(EC.presence_of_element_located((By.XPATH, '/html/body/div[5]/div/aside/div[4]/table')))

Sometimes the response I get is correct, it's a string containing the Personal Records. Other times it just grabs random info off the webpage, and occasionally it fails to find the element. I'm pretty confused as to why it works sometimes, but not all the time. Is there a better way than xpath, or am I using the wrong tag?

selenium-chromedriver web-scraping selenium python
1 Answers

Use this example how you can parse the page with beautifulsoup:

from bs4 import BeautifulSoup
import requests


url = 'https://gbr.milesplit.com/athletes/pro/5424599/stats'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

for event, time in zip(soup.select('td.event'),
                       soup.select('td.time')):
    print(event.text, time.text)

Prints:

800 Meter Run 1:48.05
1500 Meter Run 3:35.01
800 Meter Run 1:48.87
One Mile Run 3:57.02
8000 Meter Run 24:01.50
10,000 Meter Run 30:43.50
800 Meter Run 1:52.25
One Mile Run 4:05.32
1500 Meter Run 3:43.46
800 Meter Run 1:48.05
1500 Meter Run 3:35.01
800 Meter Run 1:48.87
One Mile Run 3:57.02
8000 Meter Run 24:01.50
10,000 Meter Run 30:43.50
800 Meter Run 1:52.25
One Mile Run 4:05.32
1500 Meter Run 3:43.46
user_100359850
Related