I am using an AWS Lambda function running Node v16 to generate a text file and upload it to S3. However, after writing the file to '/tmp', the file is not available. I expect this is a sync issue with my code not waiting until the file writing has completed. When I run the function a second time which reuses the existing function instance, the file is available, but I suspect it is actually the file created by the first execution.
Here is a basic code example:
exports.handler = async (event) => {
console.log('Starting')
const logFile = fs.createWriteStream('/tmp/stream-file.txt')
logFile.write('Testing...\n')
logFile.end()
const dir = await fs.promises.readdir('/tmp')
console.log(dir)
console.log('Directory Listing complete.')
};
Output from first execution:
START RequestId: 9b39370d-a765-436d-a290-617b60ae8d46 Version: $LATEST
2022-09-09T20:33:01.295Z 9b39370d-a765-436d-a290-617b60ae8d46 INFO Starting...
2022-09-09T20:33:01.298Z 9b39370d-a765-436d-a290-617b60ae8d46 INFO []
2022-09-09T20:33:01.298Z 9b39370d-a765-436d-a290-617b60ae8d46 INFO Directory Listing complete.
END RequestId: 9b39370d-a765-436d-a290-617b60ae8d46
Output from second execution:
START RequestId: 8283bdc2-7ab7-40e1-9ce8-aa57f3a06a52 Version: $LATEST
2022-09-09T20:33:34.488Z 8283bdc2-7ab7-40e1-9ce8-aa57f3a06a52 INFO Starting...
2022-09-09T20:33:34.493Z 8283bdc2-7ab7-40e1-9ce8-aa57f3a06a52 INFO [ 'stream-file.txt' ]
2022-09-09T20:33:34.493Z 8283bdc2-7ab7-40e1-9ce8-aa57f3a06a52 INFO Directory Listing complete.
END RequestId: 8283bdc2-7ab7-40e1-9ce8-aa57f3a06a52
Update
As expected, this was a sync issue and not directly related to createWriteStream. The Lambda was returning its response without waiting for the file write to complete. Here is an updated example that works. Is there a cleaner, more modern way to do this?
const fs = require('fs')
exports.handler = async function(event, context) {
return new Promise((resolve, reject) => {
console.log('Starting...')
const logFile = fs.createWriteStream('/tmp/stream-file.txt')
logFile.write('Testing...\n')
logFile.end()
logFile.on('finish', function () {
console.log('file closed.')
const dir = fs.readdirSync('/tmp')
console.log(dir)
console.log('Directory Listing.')
resolve()
})
})
}
Here is the updated output which consistently displays the newly created file:
START RequestId: dbf819af-6e8c-4e02-973e-cbdd761d566d Version: $LATEST
2022-09-12T16:24:46.213Z dbf819af-6e8c-4e02-973e-cbdd761d566d INFO Starting...
2022-09-12T16:24:46.233Z dbf819af-6e8c-4e02-973e-cbdd761d566d INFO file closed.
2022-09-12T16:24:46.235Z dbf819af-6e8c-4e02-973e-cbdd761d566d INFO [ 'stream-file.txt' ]
2022-09-12T16:24:46.235Z dbf819af-6e8c-4e02-973e-cbdd761d566d INFO Directory Listing.
END RequestId: dbf819af-6e8c-4e02-973e-cbdd761d566d