NextShardIterator never returns null when reading data from kinesis stream

user_6776224 0
Viewed 520

I am trying to read records from kinesis stream after a particular timestamp in a lambda function. I get the shards, shard iterators and then the data.
When I get the first iterator, I get the data and keep calling the same function recursively using NextShardIterator (present in the data returned). According to the documentation, the NextShardIterator will return null when there is no more data to read and it has reached $latest.
But it never returns null, and the function keeps getting invoked and eventually I get Provisioned Throughput Exceeded Exception.
I also tried using MillisBehindLatest to stop reading when the value is zero, but it also fails in some cases.

Is there a correct way to get the data from kinesis based on timestamp?

amazon-kinesis aws-lambda amazon-web-services
1 Answers

NextShardIterator will only return null when it reaches the end of a closed shard ( in cases when the shard count is updated using UpdateShardCount, SplitShard or MergeShard)

"NextShardIterator The next position in the shard from which to start sequentially reading data records. If set to null, the shard has been closed and the requested iterator does not return any more data."

If you want to start reading the stream from a specified timestamp, the best way to do this would be to use event source mapping with lambda and specifying the StartingPosition as TIMESTAMP in lambda.

user_38461860
Related