AWS Lambda Node fs.createWriteStream does not wait for file to finish writing

Viewed 34

I am using an AWS Lambda function running Node v16 to generate a text file and upload it to S3. However, after writing the file to '/tmp', the file is not available. I expect this is a sync issue with my code not waiting until the file writing has completed. When I run the function a second time which reuses the existing function instance, the file is available, but I suspect it is actually the file created by the first execution.

Here is a basic code example:


exports.handler = async (event) => {
    console.log('Starting')

    const logFile = fs.createWriteStream('/tmp/stream-file.txt')
    logFile.write('Testing...\n')
    logFile.end()

    const dir = await fs.promises.readdir('/tmp')
    console.log(dir)
    console.log('Directory Listing complete.')
};

Output from first execution:

START RequestId: 9b39370d-a765-436d-a290-617b60ae8d46 Version: $LATEST
2022-09-09T20:33:01.295Z    9b39370d-a765-436d-a290-617b60ae8d46    INFO    Starting...
2022-09-09T20:33:01.298Z    9b39370d-a765-436d-a290-617b60ae8d46    INFO    []
2022-09-09T20:33:01.298Z    9b39370d-a765-436d-a290-617b60ae8d46    INFO    Directory Listing complete.
END RequestId: 9b39370d-a765-436d-a290-617b60ae8d46

Output from second execution:

START RequestId: 8283bdc2-7ab7-40e1-9ce8-aa57f3a06a52 Version: $LATEST
2022-09-09T20:33:34.488Z    8283bdc2-7ab7-40e1-9ce8-aa57f3a06a52    INFO    Starting...
2022-09-09T20:33:34.493Z    8283bdc2-7ab7-40e1-9ce8-aa57f3a06a52    INFO    [ 'stream-file.txt' ]
2022-09-09T20:33:34.493Z    8283bdc2-7ab7-40e1-9ce8-aa57f3a06a52    INFO    Directory Listing complete.
END RequestId: 8283bdc2-7ab7-40e1-9ce8-aa57f3a06a52

Update

As expected, this was a sync issue and not directly related to createWriteStream. The Lambda was returning its response without waiting for the file write to complete. Here is an updated example that works. Is there a cleaner, more modern way to do this?

const fs = require('fs')

exports.handler =  async function(event, context) {
    return new Promise((resolve, reject) => {
        console.log('Starting...')
        
        const logFile = fs.createWriteStream('/tmp/stream-file.txt')

        logFile.write('Testing...\n')
        logFile.end()

        logFile.on('finish', function ()  {
            console.log('file closed.')
            const dir = fs.readdirSync('/tmp')
            console.log(dir)
            console.log('Directory Listing.')
            resolve()
        })

    })
}

Here is the updated output which consistently displays the newly created file:

START RequestId: dbf819af-6e8c-4e02-973e-cbdd761d566d Version: $LATEST
2022-09-12T16:24:46.213Z    dbf819af-6e8c-4e02-973e-cbdd761d566d    INFO    Starting...
2022-09-12T16:24:46.233Z    dbf819af-6e8c-4e02-973e-cbdd761d566d    INFO    file closed.
2022-09-12T16:24:46.235Z    dbf819af-6e8c-4e02-973e-cbdd761d566d    INFO    [ 'stream-file.txt' ]
2022-09-12T16:24:46.235Z    dbf819af-6e8c-4e02-973e-cbdd761d566d    INFO    Directory Listing.
END RequestId: dbf819af-6e8c-4e02-973e-cbdd761d566d
0 Answers
Related