Does setting a kinesis shard-iterator-type to LATEST risk losing messages in lambda?

user_2792630 0
Viewed 451

We are trying to determine the best shard-iterator-type for our lambda but I'm getting mixed information about the functionality of the shard iterator type AFTER a lambda has been deployed for the first time.

I have been told that if we use a shard-iterator-type of LATEST that when we go to deploy a updated version of the lambda we will loose messages since the lambda will just always pull the most recent messages from kinesis and will ignore the ones it didn't process while it was being deployed.

My question is: is this correct?

amazon-kinesis aws-lambda amazon-web-services
1 Answers

Yes, it's correct.

When your app starts reading with LATEST iterator type it will start reading from the next record coming. So all the data that has already been in the queue will be ignored. Which means that if your app has a downtime - every message during that downtime will be skipped.

You can overcome this by saving the sequence number of the latest message your app read and then using AFTER_SEQUENCE_NUMBER iterator type and provide the saved sequence number. It's like a checkpoint.

If your lambda deployed for the first time(no previous sequence number saved) you probably want to start with either:

  • TRIM_HORIZON - start by reading the oldest data in the queue. Might be a bit too much if you have a lot of data and a long retention period
  • LATEST - start reading from the next incoming message
user_11308630
Related