Parsing a date string reliably in jq

I'd like to start building up an answer here, combining the different blocks:

First, mktime takes into account DST but no other timezone information when taking a ‘broken-down time struct’:

$ TZ=Etc/Utc jq -n '[2020,6,5,7,38,57,0,186] | mktime'
1593934737
$ TZ=Europe/London jq -n '[2020,6,5,7,38,57,0,186] | mktime'
1593938337
$ TZ=America/Los_Angeles jq -n '[2020,6,5,7,38,57,0,186] | mktime'
1593938337
$ TZ=Asia/Tokyo jq -n '[2020,6,5,7,38,57,0,186] | mktime'
1593934737
$ TZ=Asia/Kathmandu jq -n '[2020,6,5,7,38,57,0,186] | mktime'
1593934737
$ unset TZ; jq -n '[2020,6,5,7,38,57,0,186] | mktime'
1593938337

Note that the only two ouputs are either 1593934737 or 1593938337, the difference of which is exactly 3600.

Second, fromdate is identical to strptime() | mktime.

Third, strflocaltime applies a time-zone offset (both permanent and DST) to unix-timestamp inputs, but not to broken-down-time inputs:

$ TZ='Europe/London' jq -n '[2020,6,5,7,38,57,0,186] | strflocaltime("%H:%M")'
"07:38"
$ TZ='Asia/Tokyo' jq -n '[2020,6,5,7,38,57,0,186] | strflocaltime("%H:%M")'
"07:38"
$ TZ='Europe/London' jq -n '1593934737 | strflocaltime("%H:%M")'
"08:38"
$ TZ='Asia/Tokyo' jq -n '1593934737 | strflocaltime("%H:%M")'
"16:38"

Fourth, now produces a unix-timestamp output which will be affected by strflocaltime's adjustment.

Going over my original confusion-causing sequence in order:

$ echo '""' | jq 'now | strftime("%H:%M")'
"18:36"        // OK, strftime is supposed to give GMT
$ echo '""' | jq 'now | strflocaltime("%H:%M")'
"19:36"        // also OK, British Summer time is one hour ahead, strflocaltime should give local time

This is explained by (3) and (4) above: now produces a unix timestamp, strflocaltime adjusts this to the local time.

$ echo '"2020-07-04T18:14:12Z"' | jq 'strptime("%Y-%m-%dT%H:%M:%SZ") | strftime("%H:%M")'
"18:14"        // strptime parses GMT, so this is fine
$ echo '"2020-07-04T18:14:12Z"' | jq 'strptime("%Y-%m-%dT%H:%M:%SZ") | strflocaltime("%H:%M")'
"18:14"        // but why is this not 19:14?!

Here, strptime produces a broken-down time which is not adjusted by strflocaltime, by (3) above.

$ echo '"2020-07-04T18:14:12Z"' | jq 'strptime("%Y-%m-%dT%H:%M:%SZ") | mktime | strftime("%H:%M")'
"19:14"        // and why does "mktime" change things around?
$ echo '"2020-07-04T18:14:12Z"' | jq 'strptime("%Y-%m-%dT%H:%M:%SZ") | mktime | strflocaltime("%H:%M")'
"20:14"       // and why does strflocaltime kick in after, but not before mktime?

strptime produces the broken-down time and mktime in theory should convert this to a unix-timestamp time assuming it is in UTC, but mktime erroneously applies the one hour DST offset (by (1) above), leading to strftime producing the (accidentally correct) local time and strflocaltime – which corrects for both permanent and DST offset (by (3) above) – giving one further (for a total of two) hours offset.

$ echo '"2020-07-04T18:14:12Z"' | jq 'fromdate | strftime("%H:%M")'
"19:14"       // I thought fromdate was synonymous to strptime?
$ echo '"2020-07-04T18:14:12Z"' | jq 'fromdate | strflocaltime("%H:%M")'
"20:14"       // I suppose this is the same issue as above with mktime

This is simply a result of (2), that fromdate uses mktime internally.

Compiling the latest commit on the master branch (a17dd32), this problem no longer appears as mktime no longer applies the one-hour offset. This is likely due to commit 3c5b1419.

As a temporary workaround, we can get the offset introduced by mktime with: jq -n 'now | gmtime | mktime - (now | trunc)'. Subtracting this offset from any occurrence of fromdate will then reliably yield UTC timestamps.

strflocaltime/1's behavior changes depending on the type of its input.

If the input is an array (a "broken down time", this is what strptime returns), strflocaltime doesn't correct it for the timezone and any seasonal time adjustments.

$ TZ=UTC jq -n '[1970,0,1,0,0,1,4,0] | strflocaltime("%H")'
"00"
$ TZ=EST jq -n '[1970,0,1,0,0,1,4,0] | strflocaltime("%H")'
"00"

But, if the input is a number (seconds since the Unix epoch, this is what mktime returns), strflocaltime feeds it to localtime first to get a broken down time; and localtime performs such corrections.

$ TZ=UTC jq -n '1 | strflocaltime("%H")'
"00"
$ TZ=EST jq -n '1 | strflocaltime("%H")'
"19"

In both cases, strftime is called with the broken down time structure, and the resulting string is returned.

This probably isn't the answer you're looking for but it might clear some things up. builtin.jq defines

def fromdateiso8601: strptime("%Y-%m-%dT%H:%M:%SZ")|mktime;
def todateiso8601: strftime("%Y-%m-%dT%H:%M:%SZ");
def fromdate: fromdateiso8601;
def todate: todateiso8601;

the following test script

#!/bin/bash
echo '"2020-07-04T18:14:12Z"' | jq -cr '
  def strptime_:  strptime("%Y-%m-%dT%H:%M:%SZ") ;
  def hour:       strftime("%H") ;
    ".                                    \(.)"
  , ". | strptime_                        \(strptime_)"
  , ". | fromdate                         \(fromdate)"  
  , ". | fromdate | todate                \(fromdate | todate)" 
  , ". | fromdate | hour                  \(fromdate | hour)"   
'

shows that on my mac (running jq 1.6) the %H strftime specifier appears sensitive to the setting of TZ.

Without explicitly setting TZ (my system's timezone is Pacific Daylight Time) I observe

bash-3.2$ ./test.sh
.                                    2020-07-04T18:14:12Z
. | strptime_                        [2020,6,4,18,14,12,6,185]
. | fromdate                         1593890052
. | fromdate | todate                2020-07-04T19:14:12Z
. | fromdate | hour                  19

explicitly setting TZ to America/Los_Angeles produces the same output

bash-3.2$ env TZ=America/Los_Angeles ./test.sh
.                                    2020-07-04T18:14:12Z
. | strptime_                        [2020,6,4,18,14,12,6,185]
. | fromdate                         1593890052
. | fromdate | todate                2020-07-04T19:14:12Z
. | fromdate | hour                  19

but explicitly setting TZ to Etc/UTC produces a different hour

bash-3.2$ env TZ=Etc/UTC ./test.sh
.                                    2020-07-04T18:14:12Z
. | strptime_                        [2020,6,4,18,14,12,6,185]
. | fromdate                         1593886452
. | fromdate | todate                2020-07-04T18:14:12Z
. | fromdate | hour                  18

I found it curious that the values from strptime are not quite the same as struct tm so digging a little deeper into builtin.c reveals some nontrivial platform-specific details along with the jv2tm which reveals the mapping from the struct tm to the json array strptime returns.

static int jv2tm(jv a, struct tm *tm) {
  memset(tm, 0, sizeof(*tm));
  TO_TM_FIELD(tm->tm_year, a, 0);
  tm->tm_year -= 1900;
  TO_TM_FIELD(tm->tm_mon,  a, 1);
  TO_TM_FIELD(tm->tm_mday, a, 2);
  TO_TM_FIELD(tm->tm_hour, a, 3);
  TO_TM_FIELD(tm->tm_min,  a, 4);
  TO_TM_FIELD(tm->tm_sec,  a, 5);
  TO_TM_FIELD(tm->tm_wday, a, 6);
  TO_TM_FIELD(tm->tm_yday, a, 7);
  jv_free(a);

  // We use UTC everywhere (gettimeofday, gmtime) and UTC does not do DST.
  // Setting tm_isdst to 0 is done by the memset.
  // tm->tm_isdst = 0;

  // The standard permits the tm structure to contain additional members. We
  // hope it is okay to initialize them to zero, because the standard does not
  // provide an alternative.

  return 1;
}

I have set the timezone to Europe/Amsterdam (+1).

With JQ 1.6:

This is expected:

$ echo '"2020-03-28T11:04:04Z"' | jq 'fromdate | strflocaltime("%H:%M (%Z)")'
$ "12:04 CET"

This is not expected:

$ echo '"2020-03-29T11:04:04Z"' | jq 'fromdate | strflocaltime("%H:%M (%Z)")'
$ "14:04 CET"

One would expect that at "2020-03-29 11:04" the time would be "13:04 CET" with summertime +1, but instead it gives me "14:04 CET"?