I am coding a program that crawls through a search query.
- scraps all titles from the first query page
- clicks each link of each title and scraps main body of the link and exits
- moves onto the next page
It works fine for most, and it did for 480 titles last night. But with the same code, it keeps returning the following.
And at different titles, so once, an error might pop up on the 8th title of the 2nd query page, and another time, an error pops up on 6th title of first page and so.
I am utterly baffled as to why it was fine before, and why it is acting the way it is now. I would appreciate some pointers.
I am working with Anaconda - Jupyter Notebook, Selenium (4.1.0.) and Python.
WebDriverException Traceback (most recent call last)
<ipython-input-100-21e892bd2f24> in <module>
1 #to_next_page(browser)
----> 2 crawl_page(browser)
3 #x = XPath_finder(10, browser)
4 #x.click()
<ipython-input-95-32c6665608a1> in crawl_page(browser)
10 # 본문 구하는 부분
11 for i, _ in enumerate(page_titles, 1):
---> 12 XPATH = XPath_finder(i, browser)
13 XPATH.click()
14 bs2 = BeautifulSoup(browser.page_source, 'html.parser')
<ipython-input-96-1904e01f1b97> in XPath_finder(i, browser)
1 def XPath_finder(i, browser):
----> 2 elements = browser.find_elements(By.CLASS_NAME, 'txt_wrap')
3 target = elements[i-1].find_element(By.TAG_NAME, 'a')
4 return target
C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in find_elements(self, by, value)
1277 # Return empty list if driver returns null
1278 # See https://github.com/SeleniumHQ/selenium/issues/4555
-> 1279 return self.execute(Command.FIND_ELEMENTS, {
1280 'using': by,
1281 'value': value})['value'] or []
C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in execute(self, driver_command, params)
422 response = self.command_executor.execute(driver_command, params)
423 if response:
--> 424 self.error_handler.check_response(response)
425 response['value'] = self._unwrap_value(
426 response.get('value', None))
C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py in check_response(self, response)
245 alert_text = value['alert'].get('text')
246 raise exception_class(message, screen, stacktrace, alert_text) # type: ignore[call-arg] # mypy is not smart enough here
--> 247 raise exception_class(message, screen, stacktrace)
248
249 def _value_or_default(self, obj: Mapping[_KT, _VT], key: _KT, default: _VT) -> _VT:
WebDriverException: Message: target frame detached
(Session info: chrome=100.0.4896.88)```