I'm using selenium python as a web scraper and then writing to a SQL database. After an iteration of the process completes, I want to open a new tab to refresh the browser and then continue scraping. Right now, the code looks somewhat like this, and works exactly as I need it to without the function refresh browser:
while True:
For i in range 30:
opentabs()
For j in range 30:
getDataFromWebpage() #pulls data from tabs and closes them, returns a list of tuples
#length 30 where each tuple contains the web data from a tab
for k in range 30:
writeToSQLDatabase()
refreshBrowser(driver, main_window) # this is the function I'm having issues with
The code for refresh browser is here:
def refreshBrowser(driver, window):
driver.execute_script("window.open('about:memory')")
driver.find_element(By.XPATH, '/html/body/div[1]/div/div[3]/button[3]').click()
driver.close()
driver.switch_to(window)
However, when it runs I receive the error message: "selenium.common.exceptions.NoSuchWindowException: Message: Browsing context has been discarded"
Ideally, this function should open a window in a new tab, do what I'm asking it to, close itslef, and then switch back to the main window so that the while loop can continue to run. The program is memory intensive, so I'm basically trying to force Firefox to delete the closed tabs' memory.
Any thoughts or suggestions?