Selenium "move_to_element" Action runs in loop

user_12075000 0
Viewed 37

I'm trying to web-scrape likes on posts on Instagram.

I've tried going through each post but after a certain number of posts Instagram stop responding to the requests.

So now I'm trying to scrape the like count without opening post, i.e when you hover your mouse on a post thumbnail it shows you the like and comment count.
Reference of thumbnail.

For this I use the move_to_element function in this manner:

1 - Search list of posts.
2 - Use move_to_element to hover on those posts.
3 - Scrape Data.
4 - Scroll
Repeat Steps 1-4.

My program goes through the first group without any problem and scrolls. But then starts going through from the 1st post and not from where it stopped or first element loaded after scrolling.

Code (Simplified):

newPost = True
while newPost:
    try:
        action =  webdriver.ActionChains(driver)
        newPost = False
        var = WebDriverWait ...
        container = driver.find_elements .. #GET THE LIST OF POSTS
        for post in container:
            action.move_to_element(post).perform()
            link = post.find_element(By.TAG_NAME, 'a').get_attribute("href")
            likes = ...
            
            if link not in postData:  #TO CHECK IF ENCOUNTERED NEW POST
                postData[link] = likes
                newPost = True 

        driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
        sleep(5)
web-scraping action selenium google-chrome python
1 Answers

The problem is that you are looping on a variable (container) which is not updated as more posts are loaded. In particular

container = driver.find_elements ...

is run before the loop, so it is not updated as you scroll down and new posts are loaded in the webpage. Let's say that when you open the webpage for the first time, 10 posts are loaded, then container will be a list of 10 webelements. When python scrolls down and more posts are loaded, they are not added to container. To solve this problem you have to use this logic

number_of_posts_to_scrape = 50
for idx in range(number_of_posts_to_scrape):
    container = driver.find_elements .. #GET THE LIST OF POSTS
    post = container[idx]
    action.move_to_element(post).perform()
    link = post.find_element(By.TAG_NAME, 'a').get_attribute("href")
    likes = ...
user_81573040
Related