sed -f not running multiple similar pattern match commands, including concatenate, against input file with multiple lines?

Viewed 68

I have a bunch of sed commands in a batch file which I run using -f.

/PATTERN1 /I,/;/s/^[ \t]*//g
/PATTERN1 /I{:a;/;/!N;s/\n/ /;ta;P;D}
s/\(PATTERN1\) \([ \tA-Za-z0-9,\"\']*\)(\(.*\))[ \t]*;[ \t]*$/\1 \2\3;/I

If I run

gsed -f sed-file.sed input-file 

it seems that since there are multiple lines which have the same pattern match in, it runs the first and ignores the others. If I comment out the lines one by one they work fine in the file on their own, but if I run them uncommented it only processes the first match.

My example input file file is

    not (this line);
pattern1 some text, ( some other text (5), some other text (6));
pattern1 this text
(
     that text (6),
     that text (7),
);
not this text either;

I would like it to look like this

    not (this line);
pattern1 some text,  some other text (5), some other text (6);
pattern1 this text that text (6), that text (7), ;
not this text either;

So, if I leave all the lines in the sed file uncommented (as above) then I get:

    not (this line);
pattern1 some text, ( some other text (5), some other text (6));
pattern1 this text (      that text (6),      that text (7), );
not this text either;

If I comment out the first 2 lines I get

    not (this line);
pattern1 some text,  some other text (5), some other text (6);
pattern1 this text
(
     that text (6),
     that text (7),
);
not this text either;

Where the first line with pattern1 in correctly has the surrounding brackets removed.

If I comment out the just the first line I get

    not (this line);
pattern1 some text, ( some other text (5), some other text (6));
pattern1 this text (      that text (6),      that text (7), );
not this text either;

Where the lines matching pattern1 are concatenated up to an including the semi-colon, but the surrounding brackets are no longer removed.

And if I comment out the last line I get the samne but the spaces are not removed...

    not (this line);
pattern1 some text, ( some other text (5), some other text (6));
pattern1 this text (      that text (6),      that text (7), );
not this text either;

And if I comment out the last 2 lines I get:

    not (this line);
pattern1 some text, ( some other text (5), some other text (6));
pattern1 this text
(
that text (6),
that text (7),
);
not this text either;

Where the spaces are correctly removed on lines begining with pattern1 and ending in semi-colon.

How can I make sure all 3 seds are processed in order, but use the one command? Or am I going to have to run them seperately?

3 Answers

When you use address range specification and then you enter a manual loop below in /PATTERN1 /I{ it conflicts with the address range.

Ex. for example:

seq 5 | sed -n '/1/,/3/{s/^/A/;p}; /1/{n;:a;/3/!{N;ba};p;}'

Each address range "remembers" if it was entered or not, and the next command gets executed anyway. If you read up until ; manually using N or n in a manual loop, then the address range will wait for the next ; to come up to stop entering.

If do the loop between PATTERN1 and ; yourself, just remove the ^[ \t]* after a newline yourself anyway.

The D deletes up to first newline in pattern space, so after you removed all the newlines s/\n/ / it will effectively delete everything.

I guess you'll want:

# if pattern is found
/PATTERN1 /I{
     # remove leading whitespaces 
     # I prefer [[:space:]]*
     s/^[ \t]*//
     # buffer everything until ';' is found
     :a; /;/!{N;ba;};
     # remove leading whitespaces after a newline
     s/\n[ \t]*/ /g; 
}
# remove the ( ... )
s/\(PATTERN1\) \([ \tA-Za-z0-9,\"\']*\)(\(.*\))[ \t]*;[ \t]*$/\1 \2\3;/I

which outputs:

    not (this line);
pattern1 some text,  some other text (5), some other text (6);
pattern1 this text  that text (6), that text (7), ;
not this text either;

which outputs:

If you have a recent GNU sed, you can run it in debug mode:

SED PROGRAM:
  /PATTERN1 /I,/;/ s/^[ \t]*//g
  /PATTERN1 /I {
    :a
    /;/! N
    s/\n/ /
    t a
    P
    D
  }
  s/\\(PATTERN1\\) \\([ \tA-Za-z0-9,\\"\\']*\\)(\\(.*\\))[ \t]*;[ \t]*$/\1 \2\3;/i

<snip>

INPUT:   'infile' line 2
PATTERN: pattern1 some text, ( some other text (5), some other text (6));
COMMAND: /PATTERN1 /I,/;/ s/^[ \t]*//g
MATCHED REGEX REGISTERS
  regex[0] = 0-0 ''

<snip>

PATTERN: pattern1 some text, ( some other text (5), some other text (6));
COMMAND:   t a
COMMAND:   P
pattern1 some text, ( some other text (5), some other text (6));
COMMAND:   D
INPUT:   'infile' line 3
PATTERN: pattern1 this text

Observe how after D, the next line is loaded into the pattern buffer, and your third command is never executed because of that. The manual says this about D (emphasis mine):

D
If pattern space contains no newline, start a normal new cycle as if the d command was issued. Otherwise, delete text in the pattern space up to the first newline, and restart cycle with the resultant pattern space, without reading a new line of input.

At this point, your pattern space never contains a newline and you just start a new cycle.

It looks like your script could be fixed like this:

/PATTERN1 /I,/;/ s/^[ \t]*//g
/PATTERN1 /I {
    :a
    /;/! N
    s/\n/ /
    t a
    s/[[:blank:]]\{1,\}/ /g
}

You don't need the P;D pattern; that's usually used when you want a moving window of multiple lines. Instead of your third command, I've added a substitution after the loop in your second command.

sed is the best tool for doing s/old/new on individual strings. What you're doing is far more complicated than that though so you shouldn't be considering using sed for it. This would produce the expected output from your posted sample input using any awk in any shell on every UNIX box:

$ cat tst.awk
tolower($0) ~ tolower("^pattern1") { inBlock = 1 }
inBlock {
    block = block $0 ORS
    if ( sub(/);\n/,";",block) ) {
        sub(/\(/,"",block)
        gsub(/[[:space:]]+/," ",block)
        print block
        block = ""
        inBlock = 0
    }
    next
}
{ print }

$ awk -f tst.awk file
    not (this line);
pattern1 some text, some other text (5), some other text (6);
pattern1 this text that text (6), that text (7), ;
not this text either;

It simply looks for a line starting with "pattern1" and when it finds it creates a block of text from there to the first ); it finds at the end of a line then removes the first ( and last ), converts all chains of white space to a single blank and prints the block. No arcane, cryptic, single character runes involved, just a clear, simple program that will run on any UNIX box and is easy to enhance in future if/when you need to do anything else.

If you don't mind using a GNU-specific solution, here's a simpler one with GNU awk that just relies on each record being terminated with a ;\n:

$ cat tst.awk
BEGIN {
    RS=ORS=";\n"
    IGNORECASE=1
}
/^pattern1/ {
    $0 = gensub(/\((.*)\)/,"\\1",1)
    gsub(/[[:space:]]+/," ")
}
{ print }

$ awk -f tst.awk file
    not (this line);
pattern1 some text, some other text (5), some other text (6);
pattern1 this text that text (6), that text (7), ;
not this text either;

If that's not all you need then post a new question that includes input that the above doesn't work for and tag it with awk. But don't keep trying to do stuff like this with sed, it's simply the wrong tool for the job.

Related