Lambda maintaining checkpoint across deployments

sumek_11439 0
Viewed 434

I've got a Lambda deployed by CloudFormation that consumes from Kinesis and produces to another Kinesis stream. The consumer of the output topic doesn't handle duplicate records well so I wanted to understand more how Lambda keeps the state of the last record/batch it consumed.

  • When redeploying lambda is the information maintaned? Or it starts reading from the StartingPosition?
  • When redeploying the Lambda via CF what difference would there be if the type of change done to the resource is Update vs Replacement? My gut feeling is that position is maintained when it's an Update, but not when it's a Replacement
  • Is information about failed items within the batch returned when ReportBatchItemFailures is enabled maintained across deployments?

Cross-posted to AWS Forum: https://forums.aws.amazon.com/thread.jspa?threadID=333823&tstart=0

amazon-kinesis aws-lambda
1 Answers

let help you from my experience of kafka/kinesis(aws).

is the checkpoint maintaned? you redeploy lambda multiple times it doesn't matter if you committed checkpoint in kinesis, As you committed till the message you read to kinesis not to lambda.

  • CheckPoints :- its essentially a mechanism allowing you to restart your process from the last checkpointed position.

coming to your 2nd question, it starts reading from the StartingPosition?

  • if you have committed checkpoint it won't start reading from start. it will start from where you had make last commit.

Does ReportBatchItemFailures make any difference?

  • yes it does matter. Suppose your lambda code processing batch of records, when batch fails to process, only records after the last successful message are retried. This reduces duplicate records processing by your application and you have more controls to handle error failures.

Hope your doubts are bit clear now :)

user_72625690
Related