AWS Firehose dynamic partitioning and date parsing

Viewed 21

I'm trying to do dynamic data partitioning by date with a kinesis delivery/firehose stream. The payload I'm expecting is JSON, with this general format

{
  "clientId": "ASGr496mndGs80oCC97mf",
  "createdAt": "2022-09-21T14:44:53.708Z",
...
}

I don't control the format of this date I'm working with.

I have my delivery firehose set to have "Dynamic Partitioning" and "Inline JSON Parsing" enabled (because both are apparently required per the AWS console UI).

I've got these set as "Dynamic Partitioning Keys"

year 
.createdAt| strptime("%Y-%m-%dT%H:%M:%S.%fZ")| strftime("%Y")
month 
.createdAt| strptime("%Y-%m-%dT%H:%M:%S.%fZ")| strftime("%m")
day
.createdAt| strptime("%Y-%m-%dT%H:%M:%S.%fZ")| strftime("%d")
hour
.createdAt| strptime("%Y-%m-%dT%H:%M:%S.%fZ")| strftime("%h")

But that gives me errors like date \"2022-09-21T18:30:04.431Z\" does not match format \"%Y-%m-%dT%H:%M:%S.%fZ.

It looks like strptime expects decimal seconds to be padded out to 6 places, but I have 3. I don't control the format of this date I'm working with. This seems to be JQ expressions, but I have exactly zero experience using it, and the AWS documentation for this stuff leaves an awful lot to be desired.

Is there a way to get strptime to successfully parse this format, or to just ignore the minute, second, and millisecond part of the time (I only care about hours)?

Is there another way to achieve what I'm trying to do here?

1 Answers

You can try following :

 .createdAt |  strptime("%Y-%m-%dT%H:%M:%S%Z") | strftime("%Y")

It is trimming the milliseconds whereas retaining rest of the information in the datetime.

Here is the jq snippet example

Related