Many resquest on GetStaticProps freezes my application build and return an server error

user_9817158 0
Viewed 313

I'm trying to list a bunch of products and I wanted to request data on node and build the page in a static way, so The homepage would be faster.

The problem is that when I make over 80 request on GetStaticProps.

The following code with 80 items, does work

const urlList = [];

for (let i = 1; i <= 80; i++) {
  const url = `myApiUrl`;
  urlList.push(url);
}

const promises = urlList.map(url => axios.get(url));
const responses = await Promise.all(promises);

return responses;

The following code with 880 items, does not work (Note that is does work outside of GetStaticProps))

const urlList = [];

for (let i = 1; i <= 880; i++) {
  const url = `myApiUrl`;
  urlList.push(url);
}

const promises = urlList.map(url => axios.get(url));
const responses = await Promise.all(promises);

return responses;

erro on console:

Uncaught     at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)

webpage error:

Server Error
Error

This error happened while generating the page. Any console logs will be displayed in the terminal window.

TLSWrap.onStreamRead
internal/stream_base_commons.js (209:20)

Is there a way to handle large requests amount like that? I'm new to hhtp requests, is there a way for me to optimize that?

next.js reactjs node.js
4 Answers

There are limits to how many connections you can create to fetch content. What you're seeing is that a method like Promise.all() isn't "smart" enough to avoid running into such limits.

Basically, when you call Promise.all() you tell the computer "do all these things simultaneously, the order does not matter, and give me all the output when done. And by the way, if a single of those operations fail stop everything and throw away all other results". It's very useful in many contexts, but perhaps not when trying to fetch over 800 things from the net..

So yes, unless you can tweak the requirements like number of allowed simultaneous connections or memory the script gets to use, you'll likely have to do this in batches. Perhaps one Promise.all() for slices of 100 jobs at a time, then next slice. You could look at using the async library and the mapLimit method or roll your own way to slice the list of jobs into batches.

user_1635490

this could be a problem based on the node version its using

but for await could also be an option for you...

user_132167970
user_59340600

As for step, for debugging purposes I would use Promise.allSettled instead of Promise.all. This should help you to understand what is the error returned by the HTTP socket. If you don't control the external API, it is likely that a firewall is blocking you from this "DDOS" attack.

As you said, batching the call doesn't solve the issue (if you queue 80 requests followed by 80 etc, you may encounter the rate limit in any case)

You should check for throttling issues, and use a module to speed limit your HTTP call like throttle-debounce

user_33094660
Related