Requirement: On insert/update to a specific table create a data stream and push that to elastic search.
Thought AWS RDS(MariaDb) ----- insert/update to say user table ---> AWS Kinesis stream -- using AWS Lambda--> AWS ES
Question How can I create a kinesis stream on insert/update to a specific table? Can I do it through AWS Lambda?
You can do this if you're using Amazon Aurora.
One of the recent updates in Amazon Aurora is the support to call a Lambda function from a stored procedure.
So, to implement your requirement, you can create an SQL trigger that calls an SQL stored procedure which then calls a Lambda function that sends that data to Elastic Search.
Reference: Invoking a Lambda Function from an Amazon Aurora DB Cluster
Note regarding transactions:
As @Ahmad Nabil pointed out in the comments below, triggers can be executed before the transaction is committed. So an SQL trigger that calls Lambda may not be what you want.
If you're not using Aurora but using MySQL or MariaDB, I recommend switching if this feature is really important to your application.
If you're using a database other than Aurora, MySQL, or MariaDB, you have to rethink your application to write the data in parallel to both RDS and ElasticSearch. I would also recommend the following architecture patterns:
I tried Using Amazon Aurora triggers to invoke a lambda function but that caused some issues when using transactions because triggers can be executed before the transaction is commited.
So what I ended up doing is modifying the application layer to invoke a lambda function asynchronously in the same "transaction" as sending data to the database, that lambda function takes a serialized JSON as the input (from the application layer) then sends that input to elasticsearch.
In case of failures in that lambda function you can do retries and use DLQ (Dead Letter Queue) to make sure data isn't lost and actually made it to elasticsearch.
