Convert text file with line breaks to json with jq

user_62612 0
Viewed 846

I've seen a lot of examples to convert a text file to json with jq, but I get stuck on something probably obvious. My input file has this format:

key1: string1
key2: string1

key1: string3
key2: string3

How can I translate that to:

[
  {"key1":"string1", "key2": "string2"},
  {"key1":"string3", "key2": "string4"}
]

I've tried to use inputs with jq, something like jq -R -n -c '[inputs|split(":")|{(.[0]):.[1]}] | add', but it fails as soon as there's a line break in the file:

jq: error (at result.txt:8): Cannot use null (null) as object key.

Thanks

jq shell json
3 Answers

Since reduction is widely understood, here's a simple reduce-based solution that does not require newlines between groups of key:value lines. That is, in constructing the array, a new object is started once a key is encountered that occurs in the previous object.

< input.txt jq -nR '
  reduce (inputs
          | select(length > 0) 
          | capture("(?<k>^[^:]*): *(?<v>.*)") 
          | {(.k):.v}) as $in (null;
    if . == null then [$in] 
    elif (.[-1] | has($in|keys_unsorted[0])) then . + [$in] 
    else .[-1] += $in end)'
user_9973580

This is a pretty good use case for a reducer.

Run with your document on its stdin, the script:

#!/usr/bin/env jq -Rncf
# ...caveat: not all platforms support more than two words on the shebang
#            you may need to stop using env and pass a full path to your jq binary!

def input_line_to_object:
  "^(?<key>[^:]+): ?(?<value>.*)$" as $re |
  (capture($re) // ("line \(.) does not parse" | halt_error(1))) | {(.key):.value};

reduce (inputs, "") as $item (
  # Initializer. .[0] == current object; .[1] == list of objects already constructed
  [{}, []];
  # Handler. Run for each line, and the final "" at the end.
  if $item == "" then
    if .[0] == {} then
      . # no new object is actually ready; maybe we had two newlines in a row?
    else
      # between objects: append .[0] to .[1], and clear .[0]
      [{}, .[1]+[.[0]]]
    end
  else
    # new data in an existing object; add new datum to .[0], leave .[1] alone
    [.[0] + ($item | input_line_to_object), .[1]]
  end
) | .[1] # take the completed list in the .[1] position

...emits as output:

[{"key1":"string1","key2":"string1"},{"key1":"string3","key2":"string3"}]
user_141220

Here's a succinct but fairly robust solution that assumes jq is called with the -n and -R options (e.g., jq -nR ...)

[foreach (inputs, null) as $in (null;
  if ($in|length) == 0
  then .out = .object | .object = null
  else .object += ($in | capture("(?<k>^[^:]*): *(?<v>.*)") | {(.k):.v})
  | .out = null
  end;
  .out // empty) ]

Here, one key point (no pun intended) is the use of capture rather than split or splits in case the value contains a colon. Note also the use of the idiom foreach (inputs,null) as .... This ensures that the last key:value input is properly handled.

user_9973580
Related