Replace newline \n with expression using sed (or awk, or tr)

user_3037237 0
Viewed 1260

I'm trying to clean up the syntax of a pseudo-json file. The file is too large to open in a text editor (20 gb), so I have to do all of this via command line (running Arch linux). The one thing I cannot figure out how to do is replace new line characters in sed (GNU sed v. 4.8)

Specifically I have data of the form:

{
    "id" : 1,
    "value" : 2
}
{
    "id" : 2,
    "value" : 4
}

And I need to put a comma after the closed curly bracket (but not the last one). So I want the output to looks like:

{
    "id" : 1,
    "value" : 2
},
{
    "id" : 2,
    "value" : 4
}

Ideally I'd just do this in sed, but from reading about this, sed flattens the text first, so it's not clear how to replace newline characters. Ideally I'd just run something like sed 's/}\n{/},\n{/g' test.json, but this doesn't work (nor does using \\n in place of \n).

I've also tried awk, but have run into similar issue of not being able to replace the combination of a hard return with brackets. And I can get tr to replace the hard returns, but not the combination of characters.

Any thoughts on how to solve this?

tr sed grep awk
5 Answers

Yeah, by default sed works line by line. You cannot match across multiple lines unless you use features to bring in multiple lines to the pattern space. Here's one way to do it, provided the input strictly follows the sample shown:

sed '/}$/{N; s/}\n{/},\n{/}' ip.txt
  • /}$/ match } at the end of a line
    • {} allows you to group commands to be executed for a particular address
    • N will add the next line to the pattern space
    • s/}\n{/},\n{/ perform the required substitution
  • Use -i option for in-place editing

This solution can fail for sequences like shown below, but I assume two lines ending with } will not occur in a row.

}
}
{
abc
}

Use sed '/}$/{N; s/}\n{/},\n{/; P; D}' if the above sequence can occur.

user_40820520

With your shown samples please try following awk program; using RS and setting its value to null then simply apply gsub(Global substitution) to substitute from }\n{ to },\n{ in matches.

awk -v RS= '{gsub(/}\n{/,"},\n{")} 1' Input_file
user_58665800

You can used GNU sed for -z to do that:

$ sed -z 's/}\n{/},\n{/g' file
{
    "id" : 1,
    "value" : 2
},
{
    "id" : 2,
    "value" : 4
}

but then it's non-portable, has to read the whole file into memory at once, and is hard to adapt if the file format isn't exactly as you expect (e.g. additional spaces, comment lines, etc.) or you need to make any additional adjustments.

I'd just use awk, e.g. using any awk in any shell on every Unix box:

awk 'NR>1{print prev (prev=="}" ? "," : "")} {prev=$0} END{print prev}' file
{
    "id" : 1,
    "value" : 2
},
{
    "id" : 2,
    "value" : 4
}

That would be portable across all Unix boxes, just reads 1 line at a time so uses almost no memory, and would be trivial to adapt for any differences in your input or additional changes you want made to the output.

user_17450010

I would use GNU AWK following way, let file.txt content be

{
    "id" : 1,
    "value" : 2
}
{
    "id" : 2,
    "value" : 4
}

then

awk 'BEGIN{RS="}\n{"}{printf "%s%s",sep,$0;sep="},\n{"}' file.txt

output

{
    "id" : 1,
    "value" : 2
},
{
    "id" : 2,
    "value" : 4
}

Explanation: I used RS (row seperator) to split on }\n{ then I do not use ORS as this would result in trailing ORS, I use trick described here.

(tested in GNU Awk 5.0.1)

user_107859750

When the last } is on the last line, you can tell sed to skip the replacement on last line

sed '$ !s/}/},/' test.json
user_32201130
Related