Is there a way to trap the output of Linux "selection" commands that do not meet their match criteria?

user_7347751 0
Viewed 49

By "selection" commands, I mean commands that do filtering such as grep, find, etc.

Background

There are at least a few different IBM mainframe environments that support pipeline processing (CMS Pipelines, for example). It's not a shell construct like it is in Bash, but usually a dedicated PIPE command that has its own built-in subcommands (stages) that perform the filtering and data processing.

One of the cooler features, in my opinion, is that "selection stages" that perform some kind of filtering usually support multiple output streams. Those data lines that meet the selection criteria are passed to the primary output stream, and if specified, those that do not are passed to a secondary output stream, where they can undergo an entirely different processing sequence.

Taking the example from the Wikipedia page linked above, which might appear in a REXX program:

'PIPE (END ?) < INPUT TXT',  /* read contents of file INPUT TXT                   */
'|A: LOCATE /Hello/',        /* find all lines containing "Hello"                 */
'|INSERT / World!/ AFTER',   /* give those to INSERT to append " World!"          */
'|B: FANINANY',              /* pass to FANINANY, accepts multiple input streams  */
'|> NEWFILE TXT A',          /* write all contents to file NEWFILE TXT A          */
'?A:',                       /* end this pipeline, 2nd output of LOCATE goes here */
'|XLATE UPPER',              /* translate text to uppercase                       */
'|B:'                        /* feed back into FANINANY stage above               */

The second occurrence of the label A: connects the second output stream of LOCATE (in this case, lines from the input file that do not match "Hello") to the input stream of XLATE, which converts the data to uppercase and passes that back to the first B: label (FANINANY). FANINANY accepts more than one input stream and will read from all connected input streams simultaneously, preserving the order of the data.

The question mark ? serves as the end character in this example, and tells the command processor "this is the end of the first pipeline" so that whatever follows can be used to independently connect another pipeline to other labeled stages, allowing you to specify the entire pipe in one command.

Example INPUT TXT file:

foo
Hello
bar
Hello
baz
Hello

After this PIPE, the NEWFILE TXT A file would then contain:

FOO
Hello World!
BAR
Hello World!
BAZ
Hello World!

Question

My main question is: is it possible to achieve something like this in Bash?

I think the framework is in place with named pipes (mkfifo, etc.) and process substitution (both of which I am familiar with).

But, the critical piece of the puzzle is: I assume that whether a Linux/UNIX command will echo all of its output to different places depends on the individual command and whether it was written to do that. If not, I suspect the code would have to be modified, after which point I could conceivably use Bash constructs to achieve this kind of thing.

pipe linux bash
2 Answers

You can easily do both transformations in a single command with GNU sed's negative match operator:

$ cd "$(mktemp --directory)"
$ cat > input.txt <<'EOF'
foo
Hello
bar
Hello
baz
Hello
EOF
$ sed '/Hello/! s/\(.*\)/\U\1/;s/\(Hello\)/\1 World!/;' input.txt
FOO
Hello World!
BAR
Hello World!
BAZ
Hello World!

I don't think there's an efficient way to send part of standard input to one command and part of it to another command (relevant question 1, 2), since any command which filters standard input will necessarily also consume all of it. Something like

while IFS= read -r line
do
    if [[ "$line" == 'Hello' ]]
    then
        echo "${line} World!"
    else
        echo "${line^^}"
    fi
done < input.txt

is unfortunately very slow.

As an alternative you can send the same file to two different input streams using COMMAND < input.txt 3< input.txt.

user_965880

Unix (/Linux) programs don't generally support multiple output streams, although with some like awk and perl you could certainly write scripts that send output to multiple places. Here's a simple awk script that sends matching lines to stdout, and non-matching ones to a named pipe:

awk -v nomatch="/path/to/pipe" '{if ($0 ~ /Hello/) {print} else {print $0 > nomatch}}'

But it sounds like you want to recombine the streams in a coherent way (i.e. keeping the line order), and there's no good way to do that, since the data flows independently through each stream (maybe at different speeds, with independent buffering etc).

But it sounds like what you really want is just a single program that transforms different lines differently, and there are a number of unix programs that'll do that. In roughly increasing order of power and flexibility (and complexity), the standard ones are sed, awk, and perl, but you can use pretty much anything (even bash itself!). Here's an example in awk:

awk '{if ($0 ~ /Hello/) {print $0 " World!"} else {print toupper($0)}}' input.txt >newfile.txt

And here's the equivalent in bash itself:

while IFS= read -r line; do
    if [[ "$line" =~ Hello ]]; then
        printf '%s World!\n' "$line"
    else
        tr "[:lower:]" "[:upper:]" <<<"$line"
    fi
done <input.txt >newfile.txt

(with newer versions of bash, you could replace that tr command with just printf '%s\n' "${line^^}")

user_898170
Related