Python Selenium - Can only use javascript on items found with xpath, but not by css-selector

Viewed 16

I was fiddling with selenium on python and I wanted to do a little program with this functionality:

  1. Make a google images query
  2. Click the 1st result
  3. Wait for the image that pops up on the right to load
  4. Screenshot said image
  5. Repeat 2 through 4 with the next n results

I'm super new to javascript and css so my approach to doing step 3 was to change either its jsname or the alt name, so in the python part of the code I can check that attribute to know when the image is loaded without some complicated python-javascript pipeline.

I tried to find the elements using css_selector since I figured it would be more portable and more robust to changes in the page layout, but for some reason I could only use javascript with the first result. The issue is not caused by reloading the image; any result beyond the first one would say something among the lines of "stale element is not in document". Which makes no sense because the path to the image doesn't change and I can read the new attributes just fine, I just couldn't use javascript on it.

Here's the relevant code:

if time.time()-last_time >= timeout:
    print('timeout - ',time.time(),last_time)
    last_time=time.time()
    n += 1
if n > max_timeout: 
    print("Couldn't load any image for",int(timeout*max_timeout),"seconds, check your connection then try again")
    driver.close()
    sys.exit(-1)
if n > last_n:
    print("Attempting download")
    filename = filename = path + query + str(n) +".png"
    last_n = n
    #This clicks the nth available image in the results. Every timeout seconds, if the image hasn't loaded, we try the next result
    click_target = 'div.isv-r:nth-child('+str(n)+')'

    driver.find_element(By.CSS_SELECTOR, click_target).click()
    #Fetch the big image after clicking the first result
    xpath = '//*[@id="Sva75c"]/div/div/div[3]/div[2]/c-wiz/div/div[1]/div[1]/div[3]/div/a/img'
    img = driver.find_element(By.XPATH, xpath)
    
    #Remove image overlay to get a clear screenshot
    xpath = '//*[@id="Sva75c"]/div/div/div[2]/a'
    element = driver.find_element(By.XPATH, xpath)
    driver.execute_script("""
        var element = arguments[0];
        element.parentNode.removeChild(element);
        """, element)
    xpath = '//*[@id="Sva75c"]/div/div/div[3]/div[2]/c-wiz/div/div[1]/div[1]/div[1]'
    element = driver.find_element(By.XPATH, xpath)
    driver.execute_script("""
        var element = arguments[0];
        element.parentNode.removeChild(element);
        """, element)
        
    #Set up an event to know when the image is done loaded
    #I don't know much javascript, so changing an attribute of the element seemed an innocuous enough method of passing information back to python
    driver.execute_script("""
        var img = arguments[0];
        if (img.complete && img.naturalHeight !== 0) {
            img.alt="KEYWORD";
        }
        else {
            img.onload = function() { img.alt="KEYWORD"; }
        }
        """, img)
        
if img.get_attribute("alt") == 'KEYWORD':
    img.screenshot(filename)
    n+=1

And these are the css_selector calls that only work on the first result

#Click the first result
driver.find_element(By.CSS_SELECTOR, 'div.isv-r:nth-child(2)').click()

#Fetch the big image that pops up on a sidebar
img = driver.find_element(By.CSS_SELECTOR, 'div a img.n3VNCb')
    
#Remove image overlay to get a clear screenshot
element = driver.find_element(By.CSS_SELECTOR, 'a.hm60ue')
driver.execute_script("""
    var element = arguments[0];
    element.parentNode.removeChild(element);
    """, element)
element = driver.find_element(By.CSS_SELECTOR, 'div.mWagE')
driver.execute_script("""
    var element = arguments[0];
    element.parentNode.removeChild(element);
    """, element)

As I said, the code works fine with xpath now, but I wanted to know why css_selector doesn't work with images except for the first result, since the path is exactly the same and it should in theory fecth the new reloaded element.

These are the full xpaths to the elements in question in order, in case it could help:

/html/body/div[2]/c-wiz/div[3]/div[2]/div[3]/div/div/div[3]/div[2]/c-wiz/div/div[1]/div[1]/div[3]/div/a/img
/html/body/div[2]/c-wiz/div[3]/div[2]/div[3]/div/div/div[2]/a
/html/body/div[2]/c-wiz/div[3]/div[2]/div[3]/div/div/div[3]/div[2]/c-wiz/div/div[1]/div[1]/div[1]
0 Answers
Related