Prettify json in a streaming log

rxyz 0
Viewed 37

I am trying to prettify aws logs stream

log format:

2022-09-07T17:14:35.987000+00:00 2022/09/07/[$LATEST]58f517bs4ebcsae033sd953bs734224 {"cold_start":true,"function_memory_size":512,"function_request_id":"a2cfdd5d-4b0b-4cfa-97c2-4af9ba6a1056","level":"ERROR","message":"Unexpected","sampling_rate":1,"timestamp":"2022-09-07T17:14:35.987Z","xray_trace_id":"1-6318d178-0e96c13a1399a74b4f353edd","extra":"{\"error\":\"AccessDeniedException: User: arn:aws:sts::\",\"stack\":\"AccessDeniedException: User: arn:aws:sts::\"}"}

I want to change this to

2022-09-07T17:14:35.987000+00:00 2022/09/07/[$LATEST]58f51004277b4ebcae033d953b734224 {
  "cold_start": true,
  "function_memory_size": 512,
  "function_request_id": "a2cfdd5d-4b0b-4cfa-97c2-4af9ba6a1056",
  "level": "ERROR",
  "message": "Unexpected",
  "sampling_rate": 1,
  "timestamp": "2022-09-07T17:14:35.987Z",
  "xray_trace_id": "1-6318d178-0e96c13a1399a74b4f353edd",
  "extra": '{"error":"AccessDeniedException: User: arn:aws:sts::","stack":"AccessDeniedException: User: arn:aws:sts::"}',
}

I tried this

aws logs tail /aws/lambda/lambda-name | sed -u -e "s/\({.*}\)/$(echo \'\\1\' | jq)/"

but I am getting this error

parse error: Invalid numeric literal at line 2, column 0

How do I do this?

sed jq logging bash json
1 Answers

If you feed your log sample to:

jq -rR 'index("{") as $ix | .[0:$ix], ( .[$ix:]|fromjson)'

you will get the log stamp followed by a copy of the corresponding valid JSON.

Your subject line indicates you want to "prettify" the JSON rather than mangle it, so I'll assume you didn't intend the latter.

If you really want the initial '{' to appear with the log stamp, then assuming a bash or bash-like shell, you could pipe the above into:

    sed $'/^[0-9][0-9][0-9][0-9]-/ {N; s/\\n{/{/; }'
peak0
Related