Puppeteer Open Dropdown Menu and then click on [1st 2nd 3rd...] option

user_12936898 0
Viewed 2321

https://www.nhtsa.gov/ratings

I have been trying to make puppeteer select an option from the dropdown menu, so I may scrape information from the website above.

1.I'm having issues declaring a correct selector for puppeteer to understand. What I mean is after telling puppeteer to click "manufacturer" at the end of the paragraph. I can't seem to click( or maybe select??) an option. The default option in this dropdown menu is select a manufacturer

2.I also would like to know how I may select the 2nd 3rd and 4th option without hard coding it in.

I haven't even begun scraping information /sad ;(

const puppeteer = require('puppeteer');

async function spider() {
    try {
        let browser = await puppeteer.launch({ headless: false});
        let page = await browser.newPage();
        await page.goto('https://www.nhtsa.gov/ratings');
        await page.click('a[data-target=".manufacturer-search-modal"]');
        await page.click('select');
        await page.click('option[value="AUDI"]');

    } catch(error) {
        console.log(error)
        await browser.close();
    }
}

export default spider
puppeteer javascript
1 Answers

I think there are a few issues. One, sometimes that page has a popup for a survey, which you may need to close first. Two, you need to wait a bit between clicking the .manufacturer-search-modal link and trying to interact with the select box, because the options in the box aren't populated immediately (maybe it's making a request to the server to get the list of options). Three, I think select boxes are a little special and clicking on them doesn't work, but you can use page.select instead. Putting that all together, plus some ugly code for selecting items by number:

async function spider() {
    try {
        let browser = await puppeteer.launch({ headless: false});
        let page = await browser.newPage();
        await page.goto('https://www.nhtsa.gov/ratings');
        try {
          // Give the 'take our survey' box a chance to pop up, and close it if it does
          await page.waitFor('.acsCloseButton', { timeout: 1000 });
          await page.click('.acsCloseButton')
        } catch {

        }
        await page.click('a[data-target=".manufacturer-search-modal"]');
        // wait for the options to be populated
        await page.waitFor('option:nth-child(2)');
        // search for AUDI
        await page.select('select', 'AUDI');
        await page.click('.manufacturer-search-submit');
        // select third element in drop-down
        // there's probably a better way to do this
        const options = await page.$$('option');
        const properties = await options[2].getProperties();
        const value = await properties.get('_value').jsonValue();
        await page.select('select', value);
        await page.click('.manufacturer-search-submit');
    } catch(error) {
        console.log(error)
        await browser.close();
    }
}
user_140491300
Related