NodeJS unzipper stream never finishes

Question

I am attempting to download a zip file, extract the contents, and push them into a database. Unfortuantely, my stream never seems to complete, so I never get the opportunity to do clean up and end the process.

I have stripped the code down to the minimum to reproduce the error.

let debugmode = false;
fs.createReadStream(zPath)
    .pipe(unzip.Parse()) 
    .pipe(Stream.Transform({
        objectMode: true,
        transform: async function(entry,e,done) {
            console.log('Item: ' + debugmode++ + ' of 819080');
            let buff = await entry.buffer();
            await entry.autodrain().promise()
            done();
        }
    }))
    .on('finish',()=>{
        console.log('DONE');
    })
    ;

The log shows the last couople of items, but never issues the word DONE.

Item: 819075
Item: 819076
Item: 819077
Item: 819078
Item: 819079
Item: 819080

Is there something I have done incorrectly? Is there something I can do to monitor for the end of file and kill the stream?

Extra Info

In the actual code, there is also a transform that reports progress based on bytes processed. There are a few bytes processed after this item.
I am using unzipper to do the extract
The zip file is a publicly accessible SEC submissions.zip. I have no problem with companies.zip. (I'm trying to find their linkable page)
I download the zip in full before processing.

Jefferey Cave · Answer

Out of frustration, I have implemented a Dead Man's Switch.

  let deadman = null;
  await new Promise((resolve)=>{
    fs.createReadStream(zPath)
      .pipe(unzip.Parse()) 
      .pipe(Stream.Transform({
        clearTimeout(deadman);
        deadman = setTimeout(resolve,60000);
        /// still do all the other stuff
      }
    }))
    .on('finish',()=>{
        clearTimeout(deadman);
        console.log('DONE');
        resolve();
    })
});

Now, every time it processes an entry, it has 60 seconds to complete processing. If it fails to complete processing in 60 seconds, it is assumed to have died and the promise is resolved. The timer is restarted every time an item is processed (the stream demonstrates it is still alive).

While I do not consider this a solution, just a work around, it is intended to be used as a single process, so it can be terminated after the run (to clean up the memory)

NodeJS unzipper stream never finishes

1 Answers