I have a use case in which I have to collect thousands of records per second from different producers and push them to elastic search using AWS firehose. I am also using a data transformation lambda on the firehose which does a fair amount of computation before passing the records back to the firehose.
Firehose is supposed to invoke lambda with each buffered batch asynchronously before the data is buffered again to be delivered to the destination.
I ran a basic test with ingress rate of 4k records per second for 15 minutes and here is how the system responded.
Looking at the firehose metrics, it is obvious that it took the firehose more than an hour to process all the incoming events. As there isn't any lambda throttles (figure 2), So I am wondering why Firehose didn't run as many lambdas as possible to be able to keep up with the input rate?
As shown in the second figure, I have around 30 lambda invocations per minute and the average processing duration is 8000ms.
So I am wondering if firehose runs lambda concurrently? and is there any firehose-lambda concurrency limits that I am missing?