JSON JQ filter by date older than bash

user_11307807 0
Viewed 1166

I have a json with this format of data in a text.json file

[
  {
    "name": "page/page1.html",
    "properties": {
      "lastModified": "2021-08-10T18:00:45+00:00",
    }
  },
  {
    "name": "page/page2.html",
    "properties": {
      "lastModified": "2021-08-10T19:24:23+00:00",
    }
  },
  {
    "name": "page/page3.html",
    "properties": {
      "lastModified": "2021-08-10T20:36:21+00:00",
    }
  }
]

I want to make a list of all the names of files which are last modified more that 30 minutes ago. This is my query at the moment to get a list of file names as a variable which i can use later.

file_names=`cat text.json | jq -r .[].name`

How can I use jq to filter for lastModified more than 30 minutes ago based on the timestamp in the properties so I only get the relevant file names?

jq bash json
3 Answers

I'd typically calculate the target date in native bash.

#!/usr/bin/env bash
# make sure we have bash new enough for printf %(...)T time-formatting
# this makes our script work even without GNU date
case $BASH_VERSION in
  ''|[123].*|4.[012].*) echo "ERROR: bash 4.3+ required" >&2; exit 1;;
esac

export TZ=UTC # force all timestamps to be in UTC (+00:00 / Z)

# faster, bash-builtin now=$(date +%s)
printf -v now '%(%s)T' -1

# faster, bash-builtin start_date_iso8601=$(date +%s -d '30 minutes ago')
start_date_epoch=$((now - 60*30))
printf -v start_date_iso8601 '%(%Y-%m-%dT%H:%M:%S+00:00)T' "$start_date_epoch"

# read our resulting names into an array (not a string)
# jq -j suppresses newlines so we can use NUL delimiters
while IFS= read -r -d '' name; do
  names+=( "$name" )
done < <(
  jq -j --arg start_date "$start_date_iso8601" '
    .[] |
    select(.properties.lastModified < $start_date) |
    (.name, "\u0000")
  ' <text.json
)

# print the content of the array we just read the names into
printf 'Matching name: %q\n' "${names[@]}"
user_141220

This seems to work

date=`date +%Y-%m-%d'T'%H:%M'Z' -d "15 min ago"`

file_names=`jq -r --arg date "$date" '.[] | select(.properties.lastModified < $date) | .name' < text.json`
user_113078070

Let jq do all date computations:

With bash 4 and above with mapfile:

mapfile -d '' last_modified < <(
  jq --join-output '(now - 1800) as $date | .[] | select((.properties.lastModified | .[:18] + "Z" | fromdate) < $date) | .name + "\u0000"' input_file.json
)

# For debug purpose
declare -p last_modified

Without mapfile, records are delimited with ASCII RS control character rather than a null byte:

IFS=$'\36' read -ra last_modified < <(jq -j '(now - 1800) as $date | .[] | select((.properties.lastModified | .[:18] + "Z" | fromdate) < $date) | .name + "\u001e"' input_file.json)

Here is the stand-alone jq script with comments:

#!/usr/bin/env -S jq -jf

# Store current timestamp minus 30 minutes (1800 seconds) as $date
(now - 1800) as $date |
 .[] |
#
select(
  (
    # Strip the numerical timezone offset out from the timestamp
    # and replace it with the Z for UTC iso8601
    # to make it an iso8601 date string that jq understands
    .properties.lastModified | .[:18] + "Z" | fromdate
  ) < $date
) |
.name + "\u0000"
user_79398710
Related