Node script causes system freeze when uploading a lot of files

Viewed 185

I have a script which is part of a larger Electron program. The script is a loop which reads files from the temp directory of the application and then uploads all files to S3. This is working well for small sets of files but on larger ones, it causes the computer to freeze and reboot with a system crash (Macbook pro 2018). A large set I have tested with consisted of about 600 files (each file small, below 1 mb).

I am not able to get any error from the script itself so it is a bit hard to debug. I can guess that this has to be with either to many fs.lstatSync or to many put active all at once.

I would like some suggestions about how I debug this (hard because the computer is freezing) or suggestions about how many open fs.lstatSync and put that is stable to have.

const upload = (): void => {
  const fileUploadPromises = [];
  const userDataPath = remote.app.getPath('temp');

  let i = 0;

  let bytes = 0;
  let totalBytes = 0;
  let lastUpdate = 0;

  for (const file of filenames) {
    const filePath = `${userDataPath}chunks/${file}.ts`;
    const stats = fs.lstatSync(filePath);
    const { size } = stats;
    const createdFile = createFileData.createFile[i];

    totalBytes = totalBytes + size;

    fileUploadPromises.push(
      put({
        url: createdFile.presignedPutUrl,
        body: fs
          .createReadStream(filePath)
          .on('data', c => {
            bytes += c.length;

            const stepSize = totalBytes / 100;

            if (bytes - lastUpdate > stepSize) {
              setProgress(bytes / totalBytes);
              lastUpdate = bytes;
            }
          })
          .on('error', e => {
            console.error(e); // eslint-disable-line
          }),
        headers: {
          'Content-length': size,
        }
      }).catch(e => {
        console.error(e); // eslint-disable-line
      })
    );

    i = i + 1;
  }
};
0 Answers
Related