how to use axios get method for every element in a very large array and save the result in json file without enobufs error?

user_10137046 0
Viewed 268

I have a very large array of length 17 thousand containing integer numbers. Now for every element in the array I want to make axios.get() request and save the response in a json file.

When I do this in the following way, I get enobuf error. My Code:

for (const id of ids) {
    axios
      .get(`${BASE_URL}/${id}?json=1`)
      .then((response) => {
          fs.writeFileSync(res)
      })
      .catch((err) => {
        console.log("error ", err);
      });
  }

My Error:

error  Error: connect ENOBUFS 69.64.69.90:443 - Local (undefined:undefined)
    at internalConnect (net.js:934:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:452:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (net.js:1077:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:73:8) {
  errno: -4060,
  code: 'ENOBUFS',
  syscall: 'connect',
  address: '69.64.69.90',
  port: 443,

I am realizing that maybe my approach is not convenient and efficient. Now my question is what my approach should be to solve this?

axios express
1 Answers

First, each of the loop iterations in this code does not wait for the result of axios call.
Secord, you need to divide the whole large array into chunks of any reasonable chunk size, for instance by using lodash.chunk method if you wish to process some number of elements simultaneously. Third, you definitely need to use asynchronous version of fs.writeFileSync, see fs.Promises

const chunks = _.chunk(ids, 10)
const writeResult = async (id) => {
  const res = await axios.get(`${BASE_URL}/${id}?json=1`)
  const fileName = id
  await fs.Promises.writeFile(fileName, res)
}
for (const chunk of chunks) {
  await Promise.all(chunk.map(x => writeResult(x))
}
user_13766180
Related