Can't exhaust the load more button to unveil all the headlines

user_9189799 0
Viewed 46

I've tried to keep clicking on more button located at the bottom of a webpage (in it's landing page) to unveil all the headlines. The thing is when I execute my script, It only click once and then stop. How can I keep clicking on that button until there is no more option to click?

Link to that website

This is my script so far:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = "https://www.newsnow.co.uk/h/Sport/Football/Championship/Transfer+News"

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get(link)

while True:
    try:
        loadmore = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "[class^='hl_more']")))
        driver.execute_script("arguments[0].scrollIntoView();",loadmore)
        loadmore.click()
    except Exception: break
driver.quit()
selenium-webdriver web-scraping selenium python-3.x python
1 Answers

Try below code to simulate required behavior:

header = driver.find_element_by_id("phead")
driver.execute_script('arguments[0].style.position = "absolute";', header)
while True:
    try:    
        loadmore = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a.hl_more.bg_tween:not(.nfloading)")))
        driver.execute_script("arguments[0].scrollIntoView();",loadmore)
        loadmore.click()
    except: break
user_45495540
Related