How to replace \n with a comma for certain lines on CLI

user_119198 0
Viewed 203

I have the following text file.

 # This is a test 1
    "watch"
    "autoconf"

    # This is another line 2
    "coreutils"
    "binutils"
    "screen"

    # This is another line 3
    "bash"
    "emacs"
    "nano"
    "bison"

    #  This is another line 4
    "libressl"
    "python"
    "rsync"
    "unzip"
    "vim"

I want to change this to the following:

# This is a test 1
watch, autoconf

# This is another line 2
coreutils, binutils, screen

# This is another line 3
bash, emacs, nano, bison

#  This is another line 4
libressl, python, rsync, unzip, vim

Remove the leading white spaces, remove quotes, replace a new line with a comma.

So far I got this:

$ cat in.txt | sed 's/"//g' | sed 's/^[[:space:]]*//'> out.txt

# This is a test 1
watch 
autoconf

# This is another line 2
coreutils
binutils
screen

# This is another line 3
bash 
emacs
nano
bison
...

I'm not sure how to replace a new line with a comma. I tried the following.

# no change
$ cat in.txt | sed 's/"//g' | sed 's/^[[:space:]]*//'| sed  's/\n/,/g'> out.txt

# changed all new lines
$ cat in.txt | sed 's/"//g' | sed 's/^[[:space:]]*//'| sed -z 's/\n/,/g'> out.txt
$ cat out.txt
# This is a test 1,watch,autoconf,,# This is another line 2,coreutils,binutils,screen,,# This is another line 3,bash,emacs,nano,bison,,#  This is another line 4,libressl,python,rsync,unzip,vim

How can I achieve this?

sed awk
5 Answers

This might work for you (GNU sed):

sed -E 's/^\s*//;/^"(\S*)"/{s//\1/;H;$!d};x;s/.//;s/\n/, /gp;$d;z;x' file

Strip off white space at the front of all lines.

Strip out double quotes and append those words to the hold space.

Otherwise, switch to the hold space, delete the first introduced newline, replace all other newlines by , , print the result and then switch back to the pattern space and print that.

user_9674920

A one-liner using GNU sed:

sed -Ez 's/\n[[:blank:]]*"?/\n/g; s/"\n([^\n])/, \1/g; s/"//g' file

or, using multiline techniques with standard sed:

sed '
    s/^[[:blank:]]*//
    /^".*"$/{
        s/.//
        s/.$//
        :a
        $b
        N
        s/\n[[:blank:]]*"\(.*\)"$/, \1/
        ta
    }' file
user_138090010

Here's an awk version. Notice that we set the record separator RS to the empty string. This tells awk to treat each block separated by an empty line as a single record. Then by setting the field separator with -F to a newline, each line in the block becomes a single field in that record.

Then it's just a matter of brute-forcing our way through the fields of each record, using sub or gsub to remove leading spaces and quotation marks, and using printf to avoid a newline when we don't want one and printing a comma instead.

$ awk -v RS="" -F'\n'  '{
  sub(/^[[:space:]]*/, "", $1);       
  print $1; 
  sep=""; 
  for (i=2; i<=NF; ++i) { 
    gsub(/[[:space:]]*"/, "", $i); 
    printf "%s%s", sep, $i; 
    sep=", " 
  } 
  print "\n"
}' file

Output:

# This is a test 1
watch, autoconf

# This is another line 2
coreutils, binutils, screen

# This is another line 3
bash, emacs, nano, bison

#  This is another line 4
libressl, python, rsync, unzip, vim
user_22292720

With your shown samples only, could you please try following. Written and tested in GNU awk.

awk '
BEGIN{
  OFS=", "
}
NF{
  gsub(/"|^ +| +$/,"")
}
/^#/ || !NF{
  if(value){
    print first ORS value
  }
  first=$0
  value=""
  if(!NF){ print }
  next
}
{
  value=(value?value OFS:"")$0
}
END{
  if(value){
    print first ORS value
  }
}
'  Input_file

Explanation: Adding detailed explanation for above.

awk '                             ##Starting awk program from here.
BEGIN{                            ##Starting BEGIN section of this program from here.
  OFS=", "                        ##Setting OFS as comma space here.
}
NF{                               ##Checking condition if line is NOT empty do following.
  gsub(/"|^ +| +$/,"")            ##Globally substituting " OR starting/ending spaces with NULL here.
}
/^#/ || !NF{                      ##Checking condition if line starts from # OR line is NULL then do following.
  if(value){                      ##Checking condition if value is NOT NULL then do following.
    print first ORS value         ##Printing first ORS value values here.
  }
  first=$0                        ##Setting first to current line here.
  value=""                        ##Nullifying value here.
  if(!NF){ print }                ##Checking condition if line is empty then simply print it.
  next                            ##next will skip all further statements from here.
}
{
  value=(value?value OFS:"")$0    ##Creating value here and keep on adding current line value to it.
}
END{                              ##Starting END block of this program from here.
  if(value){                      ##Checking condition if value is NOT NULL then do following.
    print first ORS value         ##Printing first ORS value values here.
  }
}
' Input_file                      ##Mentioning Input_file name here.
user_58665800

Using any POSIX awk in any shell on every Unix box:

$ awk -v RS= -v ORS='\n\n' -F'[[:blank:]]*\n[[:blank:]]*' -v OFS=', ' '{
    gsub(/^[[:blank:]]*|"/,"")
    printf "%s\n", $1
    for (i=2;i<=NF;i++) {
        printf "%s%s", $i, (i<NF ? OFS : ORS)
    }
}' file
# This is a test 1
watch, autoconf

# This is another line 2
coreutils, binutils, screen

# This is another line 3
bash, emacs, nano, bison

#  This is another line 4
libressl, python, rsync, unzip, vim
user_17450010
Related