Python Selenium preceding:: to find element error

user_14772220 0
Viewed 106

I'd like to

  1. find all titles in the link with '2021' in it
  2. get all links, titles and dates

I use:

all_links=[link.get_attribute('href') for link in driver.find_elements_by_xpath("//span[contains(text(),'2021'/preceding::a)]")]
titles = [title.text for title in driver.find_elements_by_xpath("//span[contains(text(),'2021') ]")]
dates = [date.text for date in driver.find_elements_by_xpath("//span[contains(text(),'2021')]/following::td/span")]
    for link, title, date in zip(links, titles, dates):
...

ERROR:SyntaxError: Failed to execute 'evaluate' on 'Document': Type conversion failed while evaluating the expression. (Session info: chrome=88.0.4324.182)

link:http://www.szse.cn/disclosure/bond/notice/index.html

Many thanks!

selenium python
3 Answers

To get the parent tag with a span that contains a text we do the following.

all_links=[link.get_attribute('href') for link in driver.find_elements_by_xpath("//a[@attachformat][.//span[contains(text(),'2021')]]")]
print(all_links)

Use waits

wait = WebDriverWait(driver, 30)
all_links=[link.get_attribute('href') for link in wait.until(EC.presence_of_all_elements_located((By.XPATH,"//a[@attachformat][.//span[contains(text(),'2021')]]")))]
print(all_links)

Outputs

['http://www.szse.cn/disclosure/listed/bulletinDetail/index.html?5bcbb736-4f52-46a0-9bd1-7ed240cb6ebc', 'http://www.szse.cn/disclosure/listed/bulletinDetail/index.html?f9aab31e-b0dc-4bda-84b3-61c099a8e341', 'http://www.szse.cn/disclosure/listed/bulletinDetail/index.html?e79765a9-e342-4798-93a0-6b53c186329d', 'http://www.szse.cn/disclosure/listed/bulletinDetail/index.html?c87ecd93-c7b6-433b-a1ce-567e1b0da785', 'http://www.szse.cn/disclosure/listed/bulletinDetail/index.html?c063e776-c96e-4ee0-aff2-a1fa8f3d14ed', 'http://www.szse.cn/disclosure/listed/bulletinDetail/index.html?1f7200a6-52f6-425b-b4bf-e87c1d4cbbba', 'http://www.szse.cn/disclosure/listed/bulletinDetail/index.html?de0aae6d-363a-49f6-af39-8976bb04d6e6', 'http://www.szse.cn/disclosure/listed/bulletinDetail/index.html?ca829df1-0ff5-4e34-9a5c-f67113e74c81', 'http://www.szse.cn/disclosure/listed/bulletinDetail/index.html?4e490995-6805-4bc2-bec1-e9417d7bf0dc', 'http://www.szse.cn/disclosure/listed/bulletinDetail/index.html?f1f000c0-1aa0-48c8-8925-c5f6e37447ed', 'http://www.szse.cn/disclosure/listed/bulletinDetail/index.html?affce6c9-2a54-4dca-b2e9-37ed3d6acfd8']

Import

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
user_99012610

the Table is created after page loaded so you have to wait, and it better to loop through the rows or tr

# wait for the table, max 10 seconds
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, '.table tbody tr'))) 

rows = driver.find_elements_by_css_selector('.table tbody tr')
for row in rows:
    title = row.find_element_by_css_selector('.title-text').text
    if '2021' not in title:
        continue
    link = row.find_element_by_css_selector('a').get_attribute('href')
    date = row.find_element_by_css_selector('.text-time').text
    print(title, link, date)
user_40823440

For links, please try with this xpath:

//span[@class='pull-left ellipsis title-text' and contains(text(), '2021')]

From the above xpath you can achieve href with //parent::a and date with //ancestor::td//following-sibling::td.

wait = WebDriverWait(driver, 20)
links = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='pull-left ellipsis title-text' and contains(text(), '2021')]")))
titles = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='pull-left ellipsis title-text' and contains(text(), '2021')]//parent::a")))
dates = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='pull-left ellipsis title-text' and contains(text(), '2021')]//ancestor::td//following-sibling::td")))

for link, title, date in zip(links, titles, dates):
    print(link.text)
    print(title.get_attribute("href"))
    print(date.text)
user_87520620
Related