How to scrape instagram post URL's using puppeteer (Node.js applicatie)

user_12231793 0
Viewed 1825

With all the changes to the current Instagram API I was trying to build a scraper. After some looking around I found puppeteer. Although it seems really straightforward I am running into a problem I can't seem to wrap my head around.

The problem is the following: I know what the div tag of a post is (.v1Nh3.kIKUG._bz0w) and how to call for it (elements = await page.$$('.v1Nh3.kIKUG._bz0w');)

If I understand the $ function correctly this should return me a promise containing an array of all the posts on 'page'.

My first question would be if this assumption is correct, and my second is how I can get the array out of. (And if that all works how to get the redirect URL contained in the child href)

puppeteer node.js javascript
2 Answers

First things first: since Instagram is a heavy javascript-powered React application, the selectors you are after may not be available right after the page is loaded. So we should wait for them to appear in the DOM:

await page.waitForSelector('.v1Nh3.kIKUG._bz0w');

Now with page.evaluate we get the posts, but since you only want the links inside of those posts, let's grab them right away in the query:

const result = await page.evaluate(() => {
    // Get elements into a NodeList
    const elements = document.querySelectorAll('.v1Nh3.kIKUG._bz0w a');
    ...
}

But we cant convert the elements from Nodelist to an Array and just return them, because they're still DOM nodes, complex unserializable objects, and they need to be serializable to be able to return from page.evaluate. So instead of returning the complete nodes we'll just get what we need: urls from href attribute:

const result = await page.evaluate(() => {
    // Get elements into a NodeList
    const elements = document.querySelectorAll('.v1Nh3.kIKUG._bz0w a');

    // Convert elements to an array, 
    // then for each item of that array only return the href attribute
    const linksArr = Array.from(elements).map(link => link.href);

    return linksArr;
});

Other ways to do it

In your question you mentioned page.$$ method. It is indeed applicable here to get handles of the objects we seek. But the code to iterate over them is not pretty:

const results = await page.$$('.v1Nh3.kIKUG._bz0w a')
for (const i in results)
{
   console.log(await(await(await results[i]).getProperty("href")).jsonValue());
}

My favourite way to get those links though would be to use page.$$eval method:

const results = await page.$$eval('.v1Nh3.kIKUG._bz0w a', links => links.map(link => link.href))

It does exactly the same what we did in page.evaluate solution but in a much more concise way.

user_27153930

In order to get elements with a certain class and return them you must use the page.evaluate method. This is an asynchronous call which returns a promise.

So, in your use case, it should look like this:

const result = await page.evaluate(() => {
    let elements = document.querySelectorAll('.v1Nh3.kIKUG._bz0w');

    let elementsArr = [];
    //Loop over elements in the array and create objects from each element 
    //with the data relevant to your logic
    for (let element of elements) {
        resultArr.push({
           //your logic
        });
    }
    return elementsArr;
});
user_106323690
Related