Authenticate using cookies for web scraping?

user_12148469 0
Viewed 796

I built an app that uses Puppeteer to scrape data from LinkedIn. I log in using email and password but would like to pass in cookies to authenticate. Here is what I currently use:

const puppeteer = require("puppeteer");

(async () => {
    try {
        const browser = await puppeteer.launch({ headless: true });
        const page = await browser.newPage();
        await page.goto("https://www.linkedin.com/login");

        await page.waitForSelector(loginBtn);
        await page.type("#username", username);
        await page.type("#password", password);
        await page.click(loginBtn, { delay: 30 });

        await browser.close();
    } catch (error) {
        console.log(`Our error = ${error}`);
    }
})();

I've seen websites like Phantombuster that use "li_at" cookies to authenticate. https://i.imgur.com/PI8fzao.png

How can I authenticate using cookies?

puppeteer web-scraping authentication cookies javascript
1 Answers

Disclaimer: I work at Phantombuster ;)

Since logging in sets a cookie in your browser on success, you can replace that step with the direct result:

await page.setCookie({ name: "li_at", value: "[cookie here]", domain: "www.linkedin.com" })

You should then be able to goto any of the website page as if you were authenticated by the login form.

user_11477720
Related