I created a database table in AWS TimeStream which has a memory retention period of 7 days:
However, when I try to upload data that is 3 days old, it will be rejected. It seems that only data newer than 24 hours is accepted:
2020-10-26T13:43:33.244+01:00: Rejected a record: The record timestamp is outside the time range [2020-10-25T12:38:33.242Z, 2020-10-26T13:18:33.242Z) of the memory store.
Is this desired, or is there a configuration option I am missing?
Here's the relevant documentation: https://docs.aws.amazon.com/timestream/latest/developerguide/storage.html
Timestream loads data into the memory store only if the timestamp is within the timespan of its retention period. So if the retention period is 1 day, the timestamp can't be more than 1 day ago.
The way I understand how changes to the memory store retention period works... it takes effect from the point of when the change is made, so if you increase the retention period from 1 day to 7 days, you can't immediately insert data with a timestamp of 7 days ago.
I'm not sure if you had increased the memory store retention period and then tried to insert old data, but it sounds like that may have been the case?
Timestream has an issue when using unix epoch timestamps instead of milliseconds. I got around this by multiplying by 1000. I read this on https://medium.com/perlego/amazon-timestream-101-3b097db680cf
I'm a bit late to the party, but I had the same issue with one of my tables on AWS TimeStream. Changing memory retention policy after creation of the table was useless, it always kept the initial retention policy of 3 months.
I had to delete and recreate the table with the new memory retention policy in order to be able to insert the data accordingly.