Wrapping delimited lines, retaining first column, with minimum final length

user_797092 0
Viewed 175

Looking to split up lines of content, retaining a headword.

I do a ton of text processing, and I like to use unix one-liners because they are easy for me to organize over time (vs. tons of scripts), I can easily chain them together, and I like (re)learning how to use classic unix functions. Often I will use a short awk, perl, or ruby one-liner, depending on which is the most elegant.

Here I have lines with X number of comma-delimited items. I want to divide these up, retaining the headword.

INPUT:

animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare, goose, horse, mouse, pig, dog, frog, bug, fish, duck, camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider, deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit, elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth, shark, salmon, shrimp, mosquito, horseshoe crab

OUTPUT:

animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab

Algorithm details:

  • input lines consist of a headword, then equals-sign, then a comma delimited list of at least 1 item.
  • In this example, most words are singles, but words could contain spaces (e.g. "horseshoe crab" at the end)
  • Split is at 9 items, UNLESS there are <3, in which case the final split could yield 12 on a line
  • There are multiple lines. e.g. the next line could be planets.

I had an idea to escape spaces, then use unix fold, and then awk to pull down the first column. This works exactly like the above:

echo "animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare, goose, horse, mouse, pig, dog, frog, bug, fish, duck, camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider, deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit, elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth, shark, salmon, shrimp, mosquito, horseshoe crab" \
| \tr ' ,' '_ ' \
| fold -s \
| perl -pe 's/=/\t/; s/^_/\t_/g;' \
| awk 'BEGIN{FS=OFS="\t"} $1==""{$1=p} {p=$1} 1' \
| tr '\t _'  '=, '

But it only considers character length (not item count), and fails to consider my special case that I don't want <3 items hanging on the final line.

I think this is an elegant little puzzle, got ideas?

perl awk ruby command-line
7 Answers

With Perl, one way

perl -wnE'
    ($head, @items) = split /\s*[,=]\s*/; 
    while (@items) { 
        @elems = splice @items, 0, 9;
        if (@elems < 3) { $lines[-1] .= ", " . join ", ", @elems }
        else            { push @lines, join ", ", @elems }
    }
    say "$head = $_" for @lines; @lines = ()
' file

or

perl -wnE'
    ($head, @items) = split /\s*[,=]\s*/; 
    push @lines, join ", ", splice @items, 0, 9  while @items; 
    $lines[-2] .= ", " . pop @lines  if 2 > $lines[-1] =~ tr/,//;
    say "$head = $_" for @lines; @lines = ()
' file

Shown over multiple lines for readability, and can be copy-pasted into a bash terminal as such, but they can also be entered on one line. Tested with an added line of 11 (9+2) items.

Notes

  • split-ing by either , or = extracts the head-word first, and then the items on a line

  • splice removes and returns (the first 9) elements, which joined by , generate a line to print, until all elements are processed. The last group is added to the previous line-to-print instead if it has fewer than 3 elements

  • In the second version all elements are processed and then the last line-to-print checked for whether it need be added to the previous one instead, by counting commas in it

user_46533790

You may consider this awk:

awk 'BEGIN {FS=OFS=" = "} {
   s = $2
   while (match(s, /([^,]+, ){1,9}(([^,]+, ){2}[^,]+$)?/)) {
      v = substr(s, RSTART, RLENGTH)
      sub(/, $/, "", v)
      print $1, v
      s = substr(s, RLENGTH+1)
   }
}' file

animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab

Pay special attention to regex used here /([^,]+, ){1,9}(([^,]+, ){2}[^,]+$)?/

That matches 1 to 9 words separated with , delimiter. This regex also has an optional part that matches upto 3 words before end of line.

user_5482250

With your shown samples only, please try following awk program. Written and tested in GNU awk should work in any awk.

Where I have created an awk variable named numberOfFields which contains number of fields you want to print(as segregated with new line as per shown samples).

awk  -v numberOfFields="9" '
BEGIN{
  FS=", ";OFS=", "
}
{
  line=$0
  sub(/ = .*/,"",line)
  sub(/^[^ ]* =[^ ]* /,"")
  for(i=1;i<=NF;i++){
    printf("%s",(i%numberOfFields==0?OFS $i ORS line" = ":\
    (i==1?line " = " $i:(i%numberOfFields>1?OFS $i:$i))))
  }
}
END{
  print ""
}
'  Input_file

OR Above code is having printf statement in 2 lines(for readability purposes) if you want to have that into a single line itself then try following:

awk  -v numberOfFields="9" '
BEGIN{
  FS=", ";OFS=", "
}
{
  line=$0
  sub(/ = .*/,"",line)
  sub(/^[^ ]* =[^ ]* /,"")
  for(i=1;i<=NF;i++){
    printf("%s",(i%numberOfFields==0?OFS $i ORS line" = ":(i==1?line " = " $i:(i%numberOfFields>1?OFS $i:$i))))
  }
}
END{
  print ""
}
'  Input_file

Explanation: Adding detailed explanation for above.

awk  -v numberOfFields="9" '            ##Starting awk program from here, creating variable named numberOfFields and setting its value to 9 here.
BEGIN{                                  ##Starting BEGIN section of awk here.
  FS=", ";OFS=", "                      ##Setting FS and OFS to comma space here.
}
{
  line=$0                               ##Setting value of $0 to line here.
  sub(/ = .*/,"",line)                  ##Substituting space = space everything till last of value in line with NULL.
  sub(/^[^ ]* =[^ ]* /,"")              ##Substituting from starting till first occurrence of space followed by = followed by again first occurrence of space with NULL in current line.
  for(i=1;i<=NF;i++){                   ##Running for loop here for all fields.
    printf("%s",(i%numberOfFields==0?OFS $i ORS line" = ":\  ##Using printf and its conditions are explained below of code.
    (i==1?line " = " $i:(i%numberOfFields>1?OFS $i:$i))))
  }
}
END{                                    ##Starting END block of this program from here.
  print ""                              ##Printing newline here.
}
'  Input_file                           ##Mentioning Input_file name here.

Explanation of printf condition above:

(
  i%numberOfFields==0                   ##checking if modules value of i%numberOfFields is 0 here, if this is TRUE:
    ?OFS $i ORS line" = "               ##Then printing OFS $i ORS line" = "(comma space field value new line line variable and space = space)
    :(i==1                              ##If very first condition is FALSE then checking again if i==1
       ?line " = " $i                   ##Then print line variable followed by space = space followed by $i
       :(i%numberOfFields>1?OFS $i:$i)  ##Else if if modules value of i%numberOfFields is greater than 1 then print OFS $i else print $i.
     )
)
user_58665800

One awk idea:

awk -F'[=,]' -v min=3 -v max=9 '
{ for (i=2; i<=NF; i++) {
      if ( (i-1) % max == 1 && (NF-i+1 > min) ) {
         if ( i > max ) print newline
         newline=$1 "="
         pfx=""
      }
      newline=newline pfx $i
      pfx=","
  }
  print newline
}
' raw.dat

Sample data:

$ cat raw.dat
animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare, goose, horse, mouse, pig, dog, frog, bug, fish, duck, camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider, deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit, elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth, shark, salmon, shrimp, mosquito, horseshoe crab
planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto, vulcan, arrakis, hoth, naboo
numbers = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
numbers2 = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

With -v min=3 -v max=9 we get:

animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab
planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto
planets = vulcan, arrakis, hoth, naboo
numbers = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
numbers2 = 1, 2, 3, 4, 5, 6, 7, 8, 9
numbers2 = 10, 11, 12, 13

Addressing OP's comment about using one-liners ...

While this awk script can certainly be jammed into a one-liner I'm guessing OP will a) find it hard to edit/maintain and b) too easy to screw up if having to (re)type over and over again.

One (obvious?) idea is to wrap the awk code in a function, eg:

splitme() {
    awk -F'[=,]' -v min=$1 -v max=$2 '
    { for (i=2; i<=NF; i++) {
          if ( (i-1) % max == 1 && (NF-i+1 > min) ) {
             if ( i > max ) print newline
             newline=$1 "="
             pfx=""
          }
          newline=newline pfx $i
          pfx=","
      }
      print newline
    }' "${3:--}"
}

NOTES:

  • parameterized the min and max values so as to pull from the command line
  • parameterized the file reference to pull from either the command line ($3) or stdin (-)
  • OP can add more logic to verify/validate input parameters as needed

Whether calling as a standalone against a file:

$ splitme 3 9 raw.dat

Or calling in a pipeline:

$ cat raw.dat | splitme 3 9

Both generate:

animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab
planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto
planets = vulcan, arrakis, hoth, naboo
numbers = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
numbers2 = 1, 2, 3, 4, 5, 6, 7, 8, 9
numbers2 = 10, 11, 12, 13
user_73661000

Here are two Ruby solutions to process one line. The variable str holds one line (the line beginning 'animals = ...' in the example).

#1 Use a regular expression

RGX = \A\w+| *= *|(?:[^,]+, *){0,10}[^,]+\z|(?:[^,]+, *){9}

def break_line(str)
  headword, _, *lines = str.scan(RGX)
  lines.each { |line| puts "#{headword} = #{line.sub(/, *\z/, '')}" }
end

brake_line(str)
animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab

The regular expression can be written in free-spacing mode to make it self-documenting.

RGX =
/
\A         # match beginning of string
\w+        # match one or more word chars (e.g., "animals")
|          # or
[ ]*=[ ]*  # "=" preceded and followed by zero or more spaces
|          # or         
(?:        # begin a non-capture group
  [^,]+    # match one or more chars other than a comma
  ,[ ]*    # match a comma and zero or more spaces
){0,10}    # end non-capture group and execute 0-10 times
[^,]+      # match one or more chars other than a comma
\z         # match end of string
|          # or
(?:        # begin a non-capture group
 [^,]+     # match one or more chars other than a comma
 ,[ ]*     # match a comma and zero or more spaces
){9}     # end non-capture group and execute 1-7 times
/x         # invoke free-spacing regex definition mode

Demo

When executed for the example str we would find the following.

headword
  #=> "animals"
_
  #=> "="
lines
  #=> ["lizard, bird, bee, snake, whale, eagle, beetle, ",
       "mule, hare, goose, horse, mouse, pig, dog, ",
       "frog, bug, fish, duck, camel, squirrel, owl, ",
       "chicken, pigeon, lion, sheep, bear, spider, deer, ",
       "tiger, lobster, dinosaur, cat, goat, rat, cricket, ",
       "rabbit, elephant, crow, fox, donkey, monkey, butterfly, ",
       "crab, leopard, moth, shark, salmon, shrimp, mosquito, horseshoe crab"]

Ruby has a convention of using the variable _ in situations when its value is not subsequently used in calculations. This is mainly to so-inform the reader.

#2 Extract and group words

def break_line(str)
  headword, *words = str.split(/ *[,=] */)
  groups = words.each_slice(9).to_a
  if groups[-1].size < 3
    groups[-2] += groups[-1]
    groups.pop
  end
  groups.each { |group| puts "#{headword} = #{group.join(', ')}" }
end

brake_line(str)
animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab

By way of a partial explanation, we would obtain the following for the example:

headword
  #=> "animals"
words
  #=> ["lizard", "bird",,..."horseshoe crab"]
groups
  #=> [["lizard", "bird", "bee", "snake", "whale", "eagle",
        "beetle", "mule", "hare"],
       ["goose", "horse", "mouse", "pig", "dog", "frog",
        "bug", "fish", "duck"],
       ["camel", "squirrel", "owl", "chicken", "pigeon", "lion",
        "sheep", "bear", "spider"],
       ["deer", "tiger", "lobster", "dinosaur", "cat", "goat",
        "rat", "cricket", "rabbit"],
       ["elephant", "crow", "fox", "donkey", "monkey", "butterfly",
        "crab", "leopard", "moth"],
       ["shark", "salmon", "shrimp", "mosquito", "horseshoe crab"]]

As the element of groups contains more than two elements (it contains five), groups is not subsequently modified. Had the last line been permitted to have at most 14 (rather than 11) elements it would have been changed to

["elephant", "crow", "fox", "donkey", "monkey", "butterfly", "crab",
 "leopard", "moth", "shark", "salmon", "shrimp", "mosquito", "horseshoe crab"]
user_2569700 
awk -F"[=,]" -v max="9" '{
        for(i=2; i<=NF; i+=max){
                row = ""
                for(j=i; j<=i+max-1; j++){
                        row=row $(j) ","
                }
                gsub(/,+$/, "", row)
                printf "%s=%s \n", $1, row
        }
    }' input_file

animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab
planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto
planets = vulcan, arrakis, hoth, naboo
numbers = 1, 2, 3, 4, 5, 6, 7, 8, 9
numbers = 10, 11, 12, 13, 14, 15, 16
cars = mercedes benz, bmw, audi, vw, porsche, seat, skoda, opel, renault
cars = mazda, toyota, honda
user_41778280

Took a while to modify my solution to make it work across both gawk and mawk by performing the equivalent of $1 = $1 towards the end of the regex chain;

$(NF!=NF=NF) expands to NF != (NF=NF) internally, which is always false, so the whole thing just means $0, but embedding $1=$1 within it :

 input ::

     1  animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare, goose, horse, mouse, pig, dog, frog, bug, fish, duck, camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider, deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit, elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth, shark, salmon, shrimp, mosquito, horseshoe crab
     2  planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto-cuz-it-shoudlve-been, planetX

 command ::

 [mg]awk '
 BEGIN {
     FS = (OFS = " = ") "*" 
   _=__ = (___="[^,]+")"[,]"
           gsub(".",_,__)
     __ = (__)_ "(("_")?("_")?"___"$)?"
 
      _ = ORS } gsub(__,"&"_ $1 OFS)+gsub("[,]"_,_)+sub((_)"+([^,]*)$","", $(NF!=NF=NF))' 

 output (mawk 1.3.4) ::

     1  animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
     2  animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
     3  animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
     4  animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
     5  animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
     6  animals = shark, salmon, shrimp, mosquito, horseshoe crab
     7  planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto-cuz-it-shoudlve-been, planetX

 output (gawk 5.1.1) ::

     1  animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
     2  animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
     3  animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
     4  animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
     5  animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
     6  animals = shark, salmon, shrimp, mosquito, horseshoe crab
     7  planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto-cuz-it-shoudlve-been, planetX
user_146721140
Related