How to use proxy with Puppeteer (page per page)

Viewed 501

Good day, there was a need to parse a large number of information from one site.

The site has about 500 thousand pages. I decided to use Puppeteer + Electron + some applied methods for this. But I ran into a problem that after a while, the site is banned by IP for an hour (not completely banned, but it is no longer possible to register an autoregue, and there is a restriction on one account). I needed to change my IP for each click on the link, or at least register the search of these proxies independently if an error occurred.

After looking at The puppeteer documentation, I realized that it gives only one opportunity - to put a proxy through the config (... args: ['--proxy-server=proxy']) and only at the start of the browser. I decided to use Addons (puppeteer-proxy, puppeteer-page-proxy, etc.) but whether because of my crookedness, or some other problem, they stubbornly did not want to work as I need (the module seems to be connected, and did not give errors, but when it came to it

So i need module to use proxy per page without restarting puppeteer instance

    let scrape = async () => {
        let browser = await puppeteer.launch({headless: false})
        let page = await browser.newPage()

        await page.setRequestInterception(true);

        page.on('request', async (request) => {
            await proxyRequest({
                page,
                proxyUrl: *my own proxy*,
                request,
            }); // doesnt working :(
        });
//parsing
0 Answers
Related