selenium error with expected string or bytes-like object

Question

I'm crawling through python selenium, but chromium automatically turns off while looking for a picture. If you don't download it, it works well. What's the problem?

This is my code

from selenium.webdriver.common.keys import Keys
import time
import urllib.request
from urllib.request import Request, urlopen



options = webdriver.ChromeOptions()
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(options=options)
driver.get("https://www.google.co.kr/imghp?hl=ko&ogbl")
elem = driver.find_element_by_name("q")
elem.send_keys("조코딩")
elem.send_keys(Keys.RETURN)
images = driver.find_elements_by_css_selector(".rg_i.Q4LuWd")
count = 1
for image in images:
    image.click()
    time.sleep(3)
    imgUrl = driver.find_element_by_css_selector(".n3VNCb").get_attribute("src")
    req = Request(imgUrl, headers={'User-Agent': 'Mozilla/5.0'})
    webpage = urlopen(req).read()
    urllib.request.urlretrieve(req, str(count) + ".jpg")
    count = count + 1



# assert "Python" in driver.title
# elem = driver.find_element_by_name("q")
# elem.clear()
# elem.send_keys("pycon")
# elem.send_keys(Keys.RETURN)
# assert "No results found." not in driver.page_source
# driver.close()

and this is my error

deprecated. Please use find_elements() instead
  images = driver.find_elements_by_css_selector(".rg_i.Q4LuWd")
c:/Users/User/Desktop/크롤링/selenium/google.py:21: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead
  imgUrl = driver.find_element_by_css_selector(".n3VNCb").get_attribute("src")
Traceback (most recent call last):
  File "c:/Users/User/Desktop/크롤링/selenium/google.py", line 24, in 
    urllib.request.urlretrieve(req, str(count) + ".jpg")
  File "C:\Users\User\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 245, in urlretrieve
    url_type, path = splittype(url)
  File "C:\Users\User\AppData\Local\Programs\Python\Python37\lib\urllib\parse.py", line 973, in splittype
    match = _typeprog.match(url)
TypeError: expected string or bytes-like object```

user_14369697 · Answer

According to the documentation for urllib.request.urlretrieve, you are passing in a url and the filename. I think you need to pass imgUrl instead of a Request object since it is looking for a string. I am also not sure why you have this line in your code as well. Typically this is a function you use the return values for. Definitely check out the documentation that I linked above for usage.

for image in images:
    image.click()
    time.sleep(3)
    imgUrl = driver.find_element_by_css_selector(".n3VNCb").get_attribute("src")
    req = Request(imgUrl, headers={'User-Agent': 'Mozilla/5.0'})
    webpage = urlopen(req).read()
    urllib.request.urlretrieve(imgUrl, str(count) + ".jpg")
    count = count + 1

selenium error with expected string or bytes-like object

1 Answers