Why is JSON from aws rds run in Docker "malformed" according to other tools?

VanillaDonuts 0
Viewed 49

To my eyes the following JSON looks valid.

{
    "DescribeDBLogFiles": [
        {
            "LogFileName": "error/postgresql.log.2022-09-14-00",
            "LastWritten": 1663199972348,
            "Size": 3032193
        }
    ]
}

A) But, jq, json_pp, and Python json.tool module deem it invalid:

# jq 1.6
> echo "$logfiles" | jq  
parse error: Invalid numeric literal at line 1, column 2

# json_pp 4.02
> echo "$logfiles" | json_pp
malformed JSON string, neither array, object, number, string or atom, 
at character offset 0 (before "\x{1b}[?1h\x{1b}=\r{...") at /usr/bin/json_pp line 51

> python3 -m json.tool <<< "$logfiles"
Expecting value: line 1 column 1 (char 0)

B) But on the other hand, if the above JSON is copy & pasted into an online validator, both 1 and 2, deem it valid.

As hinted by json_pp's error above, hexdump <<< "$logfiles" indeed shows additional, surrounding characters. Here's the prefix: 5b1b 313f 1b68 0d3d 1b7b ...., where 7b is {.

The JSON is output to a logfiles variable by this command:

logfiles=$(aws rds describe-db-log-files \
  --db-instance-identifier somedb \
  --filename-contains 2022-09-14)

# where `aws` is
alias aws='docker run --rm -it -v ~/.aws:/root/.aws amazon/aws-cli:2.7.31'

> bash --version
GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)

Have perused this GitHub issue, yet can't figure out the cause. I suspect that double quotes get mangled somehow when using echo - some reported that printf "worked" for them.

json
1 Answers

The use of docker run --rm -it -v command to produce the JSON, added some additional unprintable characters to the start of the JSON data. That makes the resulting file $logfiles invalid.

The -t option allocations a tty and the -i creates an interactive shell. In this case the -t is allowing the shell to read login scripts (e.g. .bashrc). Something in your start up scripts is outputting ansi escape codes. Often this will to clear the screen, set up other things for the interactive shell, or make the output more visually appealing by colorizing portions of the data.

James Risner0
Related