Puppeteer: Scrape sometimes works, sometimes fails with TypeError

You misuse async JavaScript a bit indeed, that causes the script to fail. For me with a slightly slow internet connection the Evaluation failed: TypeError: Cannot read property 'getAttribute' of null error was always present. You are able to improve stability a bit by replacing networkidle2 to domcontentloaded waitUntil setting in page.goto (make sure to read the docs what is the difference between them).

The main issue is the async events (communication with the chrome api) are not await-ed. You could start to refactor the script with keeping in mind the following:

Select elements more effectively

1. I recommend to use const to avoid accidental overwriting of your already selected elements.
2. Use the page context for identifying elements. Puppeteer (chrome) also provides you the $$ alias for querySelectorAll; $ : querySelector. (docs)
3. Always await async events, everything is considered async what requires communication with the chrome api!

before:

let items = document.querySelectorAll('.m-gallery-product-item-v2');

after:

const items = await page.$$('.m-gallery-product-item-v2');

Evaluate DOM content

Use elementHandles with page.evaluate to retrieve content (.getAttribute is required in very rare cases):

before:

let title = item.querySelector('h4.organic-gallery-title__outter').getAttribute("title");

after:

const title = await page.evaluate(el => el.title, (await page.$$('h4.organic-gallery-title__outter'))[i])

Do not use forEach to iterate over async events

Luckily you haven't used async/await inside your forEach loop. But, actually the lack of async was the cause your script failed if the page was not loaded in time. You do need async, just not inside forEach (no, and not inside Array.map neither!). I rather suggest to use a for...of or regular for loops if you want predictible behaviour with puppeteer actions. (In the current example the array index has a key part so I used a for loop for simplicity)

Note: It is possible to use forEach, but you will need to wrap it by Promise.all.

Use try...catch on smaller fragments of the code

E.g.: inside the loop for each iteration, so your script won't crash if only one array element has an issue. It can be very frustrating if you are running a scraper for hours and it fails near the end.

Refactor the "urls" function

The page.evaluate part keeps the code kinda-async, but you could solve this also by using the suggestions above and awaiting for each step. You'd neither return the results object in the end, but you could populate it with each iteration of the loop.

Refactored example

It won't fail anymore, also the console.log(items); will be logged to console.

const puppeteer = require('puppeteer');

/* first page search URL */
const url = keyword => `https://www.alibaba.com/trade/search?fsb=y&IndexArea=product_en&CatId=&SearchText=${keyword}`;

/* keyword to search for */
const keyword = 'future';

const results = [];

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  try {
    await page.goto(url(keyword), { waitUntil: 'domcontentloaded' });
    await page.waitForSelector('.m-gallery-product-item-v2');
    const items = await page.$$('.m-gallery-product-item-v2');

    // this console.log never gets printed to either the browser window or the terminal?
    console.log(items);

    for (let i = 0; i < items.length; i++) {
      try {
        let CurrentTime = Date.now();
        const title = await page.evaluate(el => el.title, (await page.$$('h4.organic-gallery-title__outter'))[i]);
        const link = await page.evaluate(el => el.href, (await page.$$('.organic-list-offer__img-section, .list-no-v2-left__img-container'))[i]);
        const img = await page.evaluate(el => el.getAttribute('data-image'), (await page.$$('.seb-img-switcher__imgs'))[i]);

        results.push({
          scrapeTime: CurrentTime,
          title: title,
          link: `https:${link}`,
          img: `https:${img}`
        });

      } catch (e) {
        console.error(e);
      }
    }

    console.log(results);
    await browser.close();
  } catch (e) {
    console.log(e);
    await browser.close();
  }
})();

Edit: The script can fail time-to-time due to on Alibaba's site the .organic-list-offer__img-section CSS class has been changed to .list-no-v2-left__img-container. They either AB testing two layouts with different selectors, or changing CSS classes this frequently.

Edit 2: In case of an element can have multiple selectors per user session (possibly due to product AB testing) one can use both possible selectors separated by a comma, like:

const link = await page.evaluate(el => el.href, (await page.$$('.organic-list-offer__img-section, .list-no-v2-left__img-container'))[i]);

This will ensure the element can be selected in both cases, the comma acts like an OR operator.

You need to check the existence of title, link and img before using getAttribute. Since, for example, for me, the link with your selector does not find, but it finds with this:

let link = item.querySelector('.organic-gallery-title').getAttribute('href');

I do not know what this is connected with, maybe because of different countries I have with you. In any case, you can check this selector and check how the program will work when using it. Hope this helps somehow.

You can perform an existence check as follows:

const puppeteer = require('puppeteer');
const fs = require('fs');

/* First page search URL */
const url = (keyword) => `https://www.alibaba.com/trade/search?fsb=y&IndexArea=product_en&CatId=&SearchText=${keyword}`

/* keyword to search for */
const keyword = `future`;

(async () => {
    try {
        const browser = await puppeteer.launch({
            headless: true
        });

        const page = await browser.newPage();

        await page.goto(url(keyword), { waitUntil: 'networkidle2' });

        await page.waitForSelector('.m-gallery-product-item-v2');

        const urls = await page.evaluate(() => {
            const results = [];
            const items = document.querySelectorAll('.m-gallery-product-item-v2');

            items.forEach(item => {
                const scrapeTime = Date.now();
                const titleElement = item.querySelector('h4.organic-gallery-title__outter');
                const linkElement = item.querySelector('.organic-list-offer__img-section');
                const imgElement = item.querySelector('.seb-img-switcher__imgs');
                
                /**
                 * You can combine croppedLink and link, or croppedImg and img to not make two variables if you want.
                 * But, in my opinion, separate variables are better. 
                 */
                const title = titleElement ? titleElement.getAttribute('title') : null;
                const croppedLink = linkElement ? linkElement.getAttribute('href') : null;
                const croppedImg = imgElement ? imgElement.getAttribute('data-image') : null;

                const link = croppedLink ? `https:${croppedLink}` : null;
                const img = croppedImg ? `https:${croppedImg}` : null;

                results.push({ scrapeTime, title, link, img });
            });

            return results;
        });

        console.log(urls);
        browser.close();
    } catch (e) {
        console.log(e);
        browser.close();
    }
})();