Scraping infinite scroll href with Cypress

user_1468810 0
Viewed 2764

I'm using Cypress to scrape a site with an infinite scroll.

The site is made with React, and after the user enters a search term in an input, as they scroll more products appear on the page matching the search term entered.

The code I've got so far opens a URL, navigates to the URL and collects all the hrefs that are currently visible.

I'm wondering is how I can tell cypress to scroll down further, slowly harvesting all the hrefs as it scrolls down the page, and then finally writing the hrefs to the json.

This is the code I have so far, minus the scrolling:

const arrayOfHrefs = [];

describe('Get links', () => {
  it.only('should do a product search', () => {
    cy.visit('https://www.testsite.com');
    cy.wait(5000);
    cy.get('#product_input').type('socks');
    cy.contains('socks').click(); // renders new content on the client side
    cy.wait(10000); 
    cy.get('a').each(($a) => {
      const link = $a.attr('href');
      arrayOfHrefs.push(link); // grabs all visible links and pushes them to array
    }).then(() => {
      console.log(arrayOfHrefs)
      cy.writeFile('data.json', { urls: arrayOfHrefs }) // writes array to disk
    })
  });
});
infinite-scroll cypress web-scraping
1 Answers

You did not detail what you have tried so far and what issues you're currently having regarding scrolling, but I assume scrolling down the window and then adding some logic to wait until more links become visible is sufficient. This command scrolls down the whole window to the bottom over 5000ms:

cy.scrollTo('bottom', {duration: 5000})

Note that it's not chained off from an element like:

cy.get('#some-scrollable-element').scrollTo(...)

I googled a page that has some similar dynamic infinite scroll behaviour, maybe you could base your code on the following snippet:

describe('', () => {
        before('', () => {
            cy.server()
            cy.route('GET', '**/blog/page/**').as('blog')
        })

        it('', () => {
            let numberOfChildren = 4
            cy.visit('http://www.drewleague.com/blog/')
            for (let i = 0; i < 5; i++) {
                cy.get('.posts--desktop')
                    .children()
                    .then(children => {
                        cy.wrap(children)
                            .its('length')
                            .should('eq', numberOfChildren)
                    })
             cy.scrollTo('bottom', {duration: 5000})
                 .wait('@blog')
                 .then(() => numberOfChildren += 4)
            }
        })
    })

This code scrolls down the page to the bottom 5 times, and in each iteration we check the number of children which are dynamically added, also we wait until the xhr request finishes. Not very useful on its own but you get the idea.

user_89287270
Related