efficient / safe shell-script to remove all substring matches from a delimited split string?

user_14227588 0
Viewed 170

For a string split by a delimiter, ":", what's the best way to match all SUBSTRINGS, between delimiters, that contain a different string, "XXX".

So for example to start with

/aa/:/a/b/XXX/:/bb/bb:/c/XXXd/e/f/:/cc/cc/

remove all the "XXX" containing parts -- which can be any # of instances, in any position -- to end up with

/aa/:/bb/bb:/cc/cc/

can bash do this directly? better with awk or sed?

sed awk shell bash string
3 Answers

So every entry you want to eliminate is a sequence of non-: which contains XXX, which in regex world is [^:]*XXX[^:]*

But you also want to eliminate the : that follows it, and this means you want to match and eliminate [^:]*XXX[^:]*:.

Actually, that would not allow you to eliminate a field containing XXX if it happens to be the last; to fix this, you want to match the end of line as an alternative to the closing :, so the command is

sed -E 's/[^:]*XXX[^:]*(:|$)//g' that_file

However this still has an issue: it leaves a trailing : as soon as tha last item matches. To fix this, we can simply run another substitution ad hoc, so that the full Sed command is this,

sed -E 's/[^:]*XXX[^:]*(:|$)//g;s/:$//' that_file

Do we really need to susbstitution commands?

Sed has no lookaheads, which means that whatever we match is consumed, and cannot be matched again by the same s command, even if there's the g flag.

On the other hand, we want to eliminate the fields containing XXX together with either, but not both, of the two : around it.

If we choose the right :, as I did, than it's clear that if the last field (which matches with the trailing end of line instead of :) matches XXX, it will leave the result with a trailing : (unless all fields match XXX, in which case the result is the empty string).

This means that there's no way for a single substitution command to give you the "clean" answer for all scenarios of your usecase.

Using two s commands is enough to fix this, as I've showed.

user_58252940

With GNU awk for multi-char RS and RT:

$ awk -v RS=':' '{ORS=RT} !/XXX/' file
/aa/:/bb/bb:/cc/cc/

To accommodate the case Enrico mentions in a comment below

$ cat file
/aa/:/a/b/XXX/:/bb/bb:/c/XXXd/e/f/:/cc/cc/

$ cat file1
/bb/bb:/aXXX/b/

you can do this with GNU awk for multi-char RS:

$ awk -v RS='[:\n]' '!/XXX/{printf "%s%s", sep, $0; sep=":"} END{print ""}' file
/aa/:/bb/bb:/cc/cc/

$ awk -v RS='[:\n]' '!/XXX/{printf "%s%s", sep, $0; sep=":"} END{print ""}' file1
/bb/bb

or using any awk:

$ awk -v RS=':' '!/XXX/{sub(/\n/,""); printf "%s%s", sep, $0; sep=":"} END{print ""}' file
/aa/:/bb/bb:/cc/cc/

$ awk -v RS=':' '!/XXX/{sub(/\n/,""); printf "%s%s", sep, $0; sep=":"} END{print ""}' file1
/bb/bb
user_17450010

Here is an awk to do this:

awk  '  BEGIN {FS=OFS=":"}
        {s="";
          for (i=1;i<=NF;i++) {
            if ($i~/XXX/) continue;
            s=s OFS $i
          }
          print s
        }' file
user_2986070
Related