how to get all innerHTML when only $eval gives results ($$ returns undefined)

Question

There is a table that I'm trying to pull 3 info from each row. Once done it would scroll to the bottom of the page click "load more" and then scrape the new data, and so on and so forth until there is no more "load more" button.

To pull all of the data from the table I used $$eval but that results in an undefined. However I get data if I use $eval instead but that only pulls data from the first row of the table. Why is $$eval returning "undefined" and if I can't use it how can I loop through the table to get all values with $eval?

    const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: false }); // default is true
  const page = await browser.newPage();
  await page.goto('someexamplesite.com', {
    waitUntil: 'domcontentloaded',
  });

  const ExerciseName = await page.$$eval(
    '.ExCategory-results > .ExResult-row:nth-child(2) > .ExResult-cell > .ExHeading > a',
    (e) => e.innerText
  );

  const muscleTargeted = await page.$$eval(
    ' .ExCategory-results > .ExResult-row:nth-child(2) > .ExResult-cell > .ExResult-muscleTargeted > a',
    (e) => e.innerText
  );

  const equipmentType = await page.$$eval(
    '.ExCategory-results > .ExResult-row:nth-child(2) > .ExResult-cell > .ExResult-equipmentType > a',
    (e) => e.innerText
  );

  //click on load more
  await page.waitForSelector(
    '#js-ex-content > #js-ex-category-body > .ExCategory-results > .ExLoadMore > .bb-flat-btn'
  );

  console.log({ ExerciseName, muscleTargeted, equipmentType });

    await browser.close();
})().catch((e) => {
  console.error(e);
});

code that I'm trying to scrape


    
      
        
      
    
        
          
            
                
                
            
            
              
                
                  Rickshaw Carry
                
              
              
                Muscle Targeted:
                
                  Forearms
                
              
              
                Equipment Type:
                
                  Other
                
              
            
            
              
                
                  9.6
                
                
                  Average
                
              
            
                  
        
          
            
                
                
            
            
              
                
                  Single-Leg Press
                
              
              
                Muscle Targeted:
                
                  Quadriceps
                
              
              
                Equipment Type:
                
                  Machine
                
              
            
            
              
                
                  9.6
                
                
                  Average

user_12412595 · Answer

The page.$$eval method runs Array.from(document.querySelectorAll(selector)) in the background so what you get is an array. You can't apply (e) => e.innerText on an array directly (even if it has a length of 1) without iterating it or getting the right elements by their proper indexes (e.g.: e[0].innerText), otherwise you will get undefined.

You can use an Array.map to iterate over the matching elements and collect the innerText of each into an array.

const exerciseName = await page.$$eval(
    '.ExCategory-results > .ExResult-row:nth-child(2) > .ExResult-cell > .ExHeading > a',
    elements => elements.map(el => el.innerText)
  )

Output:

[ 'Rickshaw Carry' ]

Edit:

You can iterate the row classes using a loop with an index (easiest to use a regular for loop) by (1) counting the elements with the same class names:

const rowsCounts = await page.$$eval('.ExCategory-results > .ExResult-row', rows => rows.length)

Then (2) iterate over the children .ExResult-row:nth-child(n) ..., and collect the innerTexts to an array (exerciseNames):

const exerciseNames = []
for (let i = 1; i < rowsCounts + 1; i++) { // you mignt need i = 2
  const exerciseName = await page.$eval(
    `.ExCategory-results > .ExResult-row:nth-child(${i}) > .ExResult-cell > .ExHeading > a`,
    el => el.innerText)
  exerciseNames.push(exerciseName)
}

Output:

[
  'Rickshaw Carry',
  'Single-Leg Press',
  'Landmine twist',
  'Weighted pull-up',
  'T-Bar Row with Handle',
  'Palms-down wrist curl over bench'
]

Note: The loop should be started form 1 and not 0 in such cases, as there is no "nth-child(0)". In your example the 1st is missing as well, so you might need to start the iteration at 2.

how to get all innerHTML when only $eval gives results ($$ returns undefined)

1 Answers