I am scrapping a website and the code is working is fine in headless = False mode. Basically my code is scrapping the website, looking for certain links, and clicking on them. After clicking, a new tab is opening, so the webdriver is switching to new tab, fetching the link of the new tab and storing those links. This works fine as long as headless = False. When headless = True, the url value returned by code is returning is about:blank.
options = webdriver.ChromeOptions()
options.add_argument('--disable-notifications') #Disable notification send by chrome. Ex: ALlow location access.
if headless:
options.add_argument('--headless') #Run the automation in background
driver = webdriver.Chrome(path_driver, options=options)
driver.get(website_url)
driver.set_window_size(1920, 1080)
for pos, link in enumerate(tqdm(links)):
actions = ActionChains(driver)
actions.move_to_element(link).perform() #To move the scroll bar to current element
link.click()
file_names.append(link.text)
driver.switch_to.window(driver.window_handles[1]) #Switch to new window
url = driver.current_url
file_links.append(url)
driver.close() #Close the new window
driver.switch_to.window(driver.window_handles[0]) #Switch back to old window
Can someone please help me with this?