Get text from page with selenium and python if website change location of text

user_13765015 0
Viewed 311

I am trying to get last results from a specific table called "Last matches" in this Webpage

For exemple the result of the first meeting by xpath is like this:


int1 = driver.find_element_by_xpath("//*[@id=\"sr-container\"]/div/div/div[3]/div/div/div/div[7]/div[2]/div/div/div/div/div[1]/table/tbody/tr[1]/td[5]/div/div[2]").get_attribute("innerText")

For second meeting is like this:

int2 = driver.find_element_by_xpath("//*[@id=\"sr-container\"]/div/div/div[3]/div/div/div/div[7]/div[2]/div/div/div/div/div[1]/table/tbody/tr[2]/td[5]/div/div[2]").get_attribute("innerText")

If the shape of the webpage is like the one i posted i can extract all results from that table, my problem is when i have a meeting like this Link the xpath is not the same and nothing can work

Is there a better way to locate this Last Matches table and extract data even if the shape of the page is not the same?

Help is appreciated, thanks to all of you

selenium-webdriver web-scraping selenium xpath python
3 Answers

Yes it's possible you have to introduce xpath axes :

XPATH :

//strong[text()='Last matches']/ancestor::div[contains(@class,'component-header no-margin')]/../following-sibling::div[1]/descendant::table/descendant::td[5]/div/child::div[2]/div

read more about xpath axes here

user_94440730

This was tested against both of your links...

The issue is there are 2 separate tables in the HTML (left and right) for Last matches. In order to get all of the results, you need to iterate over both of them. I used f-string below to make the xpath dynamic, since the xPath for both tables are exactly the same aside from one number between the brackets []

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.support.wait import WebDriverWait

driver = webdriver.Chrome()
driver.get("https://s5.sir.sportradar.com/sports4africa/en/1/season/82128/headtohead/613958/33714/match/27197856")

tables = [1,2]
results = []
for table in tables:
    last_match_table = f"(//table[@class='table'])[{table}]//tbody/tr"
    scores = WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.XPATH,(last_match_table))))
    for score in scores:
        results.append(score.get_attribute("innerText"))

for row in results:
    text_split = row.split()
    final = ' '.join(text_split[4:])
    print(final)

Note that I also used a much more generic xPath. When changes happen in the DOM (as you saw) this won't be affected. There are 4 tables on that page given this path //table[@class='table'], 2 for Last matches and 2 for Next matches, so we only want to target the first 2, hence iterating over the list tables = [1,2] dynamically to populate the xPaths

Results:

Bolivar 2:0 CD Real Tomayapo
CD Real Tomayapo 2:1 Blooming
Guabira 0:2 CD Real Tomayapo
CD Real Tomayapo 0:0 Real Potosi
Royal Pari 4:2 CD Real Tomayapo
CD Real Tomayapo 1:0 Always Ready
Aurora 3:0 Independiente Petrolero
Aurora 1:1 Bolivar
Blooming 1:0 Aurora
Aurora 2:1 Guabira
Real Potosi 1:1 Aurora
Aurora 0:8 Royal Pari
user_44390190

One more good option is to use ancestor in xpath. I bound the main locator to the table name, it would be more reliable.

Using it you can find other locators and their text. Just put them to the loop with correct paths. In the child xpath .//td means a direct child of the main locator with element name td.

My solution:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


url = 'https://s5.sir.sportradar.com/sports4africa/en/1/season/80526/headtohead/334075/340986/match/27195664'
driver = webdriver.Chrome(executable_path='/snap/bin/chromium.chromedriver')
driver.get(url)
driver.implicitly_wait(10)
WebDriverWait(driver, 15).until(EC.presence_of_all_elements_located((By.XPATH, "//strong[text()='Last matches']/ancestor::div[6]//tbody/tr")))
rows= driver.find_elements_by_xpath("//strong[text()='Last matches']/ancestor::div[6]//tbody/tr")
output = []
for res in rows:
    score = res.find_element_by_xpath(".//td[5]//div[@class=' no-wrap']").get_attribute("innerText")
    output.append(score)
print(output)

Output:
First link: ['0:4', '3:4', '2:2', '0:1', '3:0', '2:2', '0:4', '1:0', '2:1', '1:1', '1:2', '2:4']

Second link: ['2:0', '2:1', '0:2', '0:0', '4:2', '1:0', '3:0', '1:1', '1:0', '2:1', '1:1', '0:8']

Update: The fastest way to swap scores I could do is with getting two scores separately, placing them in a separate lists and swapping using zip. The results are two lists of tuples.

first_score = []
second_score = []
for res in rows:
    first = res.find_element_by_xpath(".//td[5]//div[@class=' no-wrap']/div[1]").get_attribute("innerText")
    first_score.append(first)
    second = res.find_element_by_xpath(".//td[5]//div[@class=' no-wrap']/div[3]").get_attribute("innerText")
    second_score.append(second)
first_list = list(zip(first_score, second_score))
second_list = list(zip(second_score, first_score))
print(first_list)
print(second_list)

The results are two lists of tuples.

[('0', '4'), ('3', '4'), ('2', '2'), ('0', '1'), ('3', '0'), ('2', '2'), ('0', '4'), ('1', '0'), ('2', '1'), ('1', '1'), ('1', '2'), ('2', '4')]
[('4', '0'), ('4', '3'), ('2', '2'), ('1', '0'), ('0', '3'), ('2', '2'), ('4', '0'), ('0', '1'), ('1', '2'), ('1', '1'), ('2', '1'), ('4', '2')]

There are more effective ways to do it, but I'd suggest to ask a separate question for it.

user_127301120
Related