Remove trailing json comma with command line tools

Since the input seems to be some kind of extension of JSON, you could use a command-line tool intended for such extensions. For example:

$ hjson -j < input.txt

or:

$ any-json --input-format=hjson input.txt

Output in both cases

{
  "key1": "value1",
  "object": {
    "key2": "value2"
  },
  "key3": "value3"
}

Using the hold buffer:

sed '/^ *\}/{H;x;s/\([^}]\),\n/\1\n/;b};x;/^ *}/d' input

This is just a sed exercise, I don't think sed is the right tool for this job. It also needs a newline at the end or that the file ends with a }.

Not an answer with sed but a (python) solution:

# load as python dictionary
d = {
  "key1": "value1",
  "object": {
    "key2": "value2",
  },
  "key3": "value3",
}

import json

json.dumps(d) # valid json string

Here is one in GNU awk. It uses " as field separator and removes commas before [ \n]*} from odd fields (outside quotes, will probably fail for "escaped \" inside"). Added "key4": "value4,}", to the file:

$ cat file
{
  "key1": "value1",
  "object": {
    "key2": "value2",
  },
  "key3": "value3",
  "key4": "value4,}",
}

The script processes the whole file as a single record (RS="^$") so it might not work for big files as-is:

$ awk '
BEGIN {
    FS=OFS="\""
    RS="^$"
}
{
    for(i=1;i<=NF;i++) {                         # or i+=2 and remove the if
        if(i%2)
            $i=gensub(/,([ \n]*\})/,"\\1","g",$i)
    }
}1' file

Output:

{
  "key1": "value1",
  "object": {
    "key2": "value2"
  },
  "key3": "value3",
  "key4": "value4,}"
}

sed and awk commands wasn’t working for me, so I ended up writing a small remove-json-trailing-comma.js file

const { readFileSync, writeFileSync } = require("fs");

const regex = /,(?!\s*?[{["'\w])/g;

const files = process.argv.splice(2);
for (let file of files) {
    const input = readFileSync(file).toString();
    let correct = input.replace(regex, '');
    writeFileSync(file, correct);
}

And then I could use it like this node remove-json-trailing-comma.js file.json (and work also with multiples files like node remove-json-trailing-comma.js **/package.json)

https://github.com/stedolan/jq/wiki/FAQ#processing-not-quite-valid-json seems to be your answer

# this works
echo '{"a": 1,}' | jq -n -f /dev/stdin

# as well as this
cat <<EOF | jq -n -f /dev/stdin
{
  "key1": "value1",
  "object": {
    "key2": "value2",
  },
  "key3": "value3",
}
EOF

I got it by loading the json as yaml with python pyyaml library and worked fine.

So for this example:

$ echo '{"a": 1,}' | jq
parse error: Expected another key-value pair at line 1, column 9

pyyaml fixes the input:

$ echo '{"a": 1,}' | python3 -c "import sys, json, yaml; print(json.dumps(yaml.safe_load(sys.stdin)))" | jq

{
  "a": 1
}

A more complex example:

$ echo '{"a": [1,],}' | python3 -c "import sys, json, yaml; print(json.dumps(yaml.safe_load(sys.stdin)))" | jq

{
  "a": [
    1
  ]
}