Querying a GSI immediately after DynamoDB streams event received - is it reliable?

Viewed 421

From the below quote, I struggle to understand whether the propagation of a transaction to global secondary indexes (GSIs), streams, and backups happen in parallel, or is sequential.

Once a transaction completes, the changes made within that transaction are propagated to global secondary indexes (GSIs), streams, and backups.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transaction-apis.html

Hence my question: Is a dynamodb stream event raised only after all GSIs are rebuilt(replicated) or such a statement is false? In other words, can I be sure that I can find a projection for an item in some GSI if I query the GSI with the corresponding keys, immediately after I receive a stream event for its INSERT

To give some context - I am working on a concept of a procedure that can span across multiple lambdas, and each lambda is doing only one transaction. Prior to starting the procedure I know exactly how many transactions I should expect, and I am counting them with the help of dynamo streams and atomic counters, this way I can determine when a procedure has finished(edit Please do not redirect me to using step functions, I know their great potential but my stack will explode :))

However, if the above quote from AWS does not guarantee GSI replication prior to issuing the dynamo stream event, I am a bit screwed, because I can not immediately start other procedures that rely on the data inserted but want to query for it via a GSI...

Thanks!

edit: After helpful discussions and some more digging, I found this aws article that provides me with further understanding, particularly the Best practices for working with DynamoDB Streams section

2 Answers

Stream events and GSI replication occur simultaneously. There is no guarantee one will occur before the other. With properly configured GSI's that are not experiencing any back pressure from hot partitions replication will occur in less than 10ms. Stream event processing typically occurs in less than 1 second. With a well designed data model it is highly likely that GSI replication will complete before a Stream event is processed, but there is no guarantee that this will be the case.

Yes, it's possible. GSIs and Streams are written in parallel, but also asynchronously. The timing varies, but it's typically on the order of tens to hundreds of milliseconds for both. As a design goal, the propagation should happen within 1s, but there are outliers.

Related