I am trying to implement a serverless MapReduce using AWS Lambda functions for small payloads and low latency. I want to use multiple lambda functions for the Map, and another lambda function for the Reduce. Each Map function is expected to take less than a second to complete, and to produce a payload smaller than 100KB per function. I also want the Reduce function to be started at the same time as the Map functions, so that it can start reducing map results as soon as the first ones become available.
My question is the following: what is the best way to send notifications and payloads from the Map functions to the Reduce function, knowing that a single Reduce function must receive notification and payloads from all the Map functions during its lifespan (about 5 seconds)?
The Reduce function could poll an SQS queue, but I need to run multiple MapReduce jobs in parallel, and I am concerned by the latency resulting from the creation of one dedicated SQS queue for each MapReduce job. Is there a better way to approach this problem?