NodeJS fs.unlinkSync not freeing up hard-drive space on aws ec2 server

user_3935218 0
Viewed 355

I have a NodeJS application on Elastic Beanstalk.

I'm downloading a large number of large zips, opening and processing them, then deleting the zip files via fs.unlinkSync

When testing I see the files get deleted, but apparently unlink won't actually deallocate the space while a process is using it?

How can I instruct fs to free up the memory allocation?

I am writing the file via

let writeStream = fs.createWriteStream(file.name);
writeStream.on('finish', () => {
  // a bunch of logic that eventually returns a promise

and inside its resolve/reject

fs.unlinkSync(file.name)

I've gone through and those are the only times I'm touching the file, everything else is closed. Is there some other event I need to wait for for the writeStream to close?

-- to fully explain the workflow

for (let i = 0; i < records.length; i++) {
  await processZip(records[i]);
  fs.unlinkSync(records[i].name);
}

inside the processZip method

return new Promise(async (resolve, reject) => {

  var readStream = this.s3Access.getObjectStream(guid);
  let writeStream = fs.createWriteStream(file.name);
  readStream.pipe(writeStream);

  writeStream.on('finish', () => {
    resolve();
  }
}
node.js
1 Answers

According to NodeJS docs unlinkSync() is calling C/C++ API call unlink.

Here is what docs says here

unlink() deletes a name from the filesystem. If that name was the last link to a file and no processes have the file open, the file is deleted and the space it was using is made available for reuse.

If the name was the last link to a file but any processes still have the file open, the file will remain in existence until the last file descriptor referring to it is closed.

If the name referred to a symbolic link, the link is removed.

If the name referred to a socket, FIFO, or device, the name for it is removed but processes which have the object open may continue to use it.

Basically all links should be closed for a file to be deleted.

Try to think of other processes that may use the files you are trying to delete (files opened with fs.open/openSync() which were not closed etc.)

UPDATED

It looks like the file is deleted after the promise returned by processZip is resolved successfully.

It would be useful to also delete the file if the promise was rejected (i.e. error thrown in async function).

for (let i = 0; i < records.length; i++) {
  try {
    await processZip(records[i]);
  } finally {
    // delete file after success or error
    fs.unlinkSync(records[i].name);
  }
}

Additionally 'error' event handlers can be added to both streams in processZip.

I've found this note in stream.pipe() docs, maybe it's related to the issue:

One important caveat is that if the Readable stream emits an error during processing, the Writable destination is not closed automatically. If an error occurs, it will be necessary to manually close each stream in order to prevent memory leaks.

return new Promise(async (resolve, reject) => {

  var readStream = this.s3Access.getObjectStream(guid);
  let writeStream = fs.createWriteStream(file.name);
  readStream.pipe(writeStream);

  writeStream.on('finish', () => {
    resolve();
  }

  readStream.on('error', err => {
    readStream.end();
    writeStream.end();
    reject(err);
  }

  writeStream.on('error', err => {
    readStream.end();
    writeStream.end();
    reject(err);
  }
}
user_145968560
Related