Using SED to modify the same string twice and delete it

Question

I am in the situation where I have a text file and I need to find the string that matches a specific pattern and modify it twice to have two different outputs, something like the following:

let's say this is the original file:

ID_00:/hdfs_01              name1

ID_01:/hdfs_02              name2

ID_02:/hdfs_03              name3

ID_03:/hdfs_app_data_01     name4

ID_04:/hdfs_app_data_02     name5

ID_05:/hdfs_cmmd_prt        name6

you will find that it is similar to the structure of an fstab.

And the expected result is :

ID_00:/hdfs_01              name1_old
ID_06:/hdfs_01              name1

ID_01:/hdfs_02              name2_old
ID_07:/hdfs_02              name2

ID_02:/hdfs_03              name3_old
ID_08:/hdfs_03              name3

ID_03:/hdfs_app_data_01     name4_old
ID_09:/hdfs_app_data_01     name4

ID_04:/hdfs_app_data_02     name5_old
ID_10:/hdfs_app_data_02     name5

ID_05:/hdfs_cmmd_prt        name6_old
ID_11:/hdfs_cmmd_prt        name6

So far, I've considered doing a simple sed search for hdfs as is the only thing in common, add it to a different text file, modify it, and then adding the entire modified block back to the source file.

Do this twice, once to add the old suffix and once to modify the ID.

Something like:

sed -n '/hdfs/p' >> new.file

Add the old suffix in the new.file with sed/awk save the result in a different file.

Now do it again to update the ID and save the result in a different file.

Then with those 2 files created with the expected output I can easily remove the record on the current file, add the new ones and reload the app.

this is more or less the plan so far.

But I would like to know if this could be achieved using only sed and its buffer. Saving the original matched pattern, modify it once for the old suffix and print it [this is output #1] and then modify the original line again for the ID and print it again [this is output #2 ], then we can remove the original line.

UPDATE:

The ID_### is a fixed value. Has no progression and is not captured arithmetically. In the example, it seems to have progression but it doesn't.

Then it should look something like this:

ID_00:/hdfs_01              name1_old
ID_25:/hdfs_01              name1

ID_01:/hdfs_02              name2_old
ID_10:/hdfs_02              name2

ID_02:/hdfs_03              name3_old
ID_56:/hdfs_03              name3

The value is selected as per the maatching string.

Where, for example: hdfs_01 will definitely carry the ID ID_25, but /hdfs_02 will carry the ID_10.

This value will be determined at run time.

We also have some other values inside that we should not modify like:

ID_00:/hdfs_01              name1

ID_01:/hdfs_02              name2

ID_20:/DSF_03               name3
 
ID_21:/TEST_05              name3

So the result will look like:

ID_00:/hdfs_01              name1_old
ID_25:/hdfs_01              name1

ID_01:/hdfs_02              name2_old
ID_10:/hdfs_02              name2

ID_20:/DSF_03               name3
 
ID_21:/TEST_05              name3

htaccess · Answer

The following awk 1 liner does the job:

awk -F":" '/hdfs/{ split($1, a, /_/); print $0 "_old" "
" "ID_"  sprintf("%02d", a[2] + 6) ":" $2 "
" }' file
ID_00:/hdfs_01              name1_old
ID_06:/hdfs_01              name1

ID_01:/hdfs_02              name2_old
ID_07:/hdfs_02              name2

ID_02:/hdfs_03              name3_old
ID_08:/hdfs_03              name3

ID_03:/hdfs_app_data_01     name4_old
ID_09:/hdfs_app_data_01     name4

ID_04:/hdfs_app_data_02     name5_old
ID_10:/hdfs_app_data_02     name5

ID_05:/hdfs_cmmd_prt        name6_old
ID_11:/hdfs_cmmd_prt        name6

markp-fuso · Answer

Assumptions:

lines of interest start with ID_##: and have no trailing white space (all other lines will be printed as is without any modifications)
the ## is a zero-padded, 2-digit number
the number of lines starting with ID_##: could vary and is not known beforehand so ...
we keep a count (c) of the number of lines starting with ID_##:
for new lines we add c to ## to generate a new ID_##: label (eg, ## = 02 and c = 6 so new label will be ID_08:)

I don't know sed well enough to know if a running count can be maintained and then used to 'add' to a string in the pattern space (assuming the objective is to use a single sed script) so since the question is also tagged with awk ...

One (verbose) awk idea:

awk '
    { lines[NR]=$0
      if ($0 ~ /^ID_/) c++
    }
END { for (i=1;i<=NR;i++) {
          if (lines[i] ~ /^ID/) {
             print lines[i] "_old"                 # assumes line has no trailing white space otherwise we end up with "_old"
             split(lines[i],a,":")
             split(a[1],b,"_")
             newid=sprintf("%02d",b[2]+c)
             print "ID_" newid ":" a[2]
          }
          else
             print lines[i]
      }
    }
' old.file

This generates:

ID_00:/hdfs_01              name1_old
ID_06:/hdfs_01              name1

ID_01:/hdfs_02              name2_old
ID_07:/hdfs_02              name2

ID_02:/hdfs_03              name3_old
ID_08:/hdfs_03              name3

ID_03:/hdfs_app_data_01     name4_old
ID_09:/hdfs_app_data_01     name4

ID_04:/hdfs_app_data_02     name5_old
ID_10:/hdfs_app_data_02     name5

ID_05:/hdfs_cmmd_prt        name6_old
ID_11:/hdfs_cmmd_prt        name6

NOTE: I don't understand OP's last part of the question: then we can remove the original line.; for now I've matched what OP listed as the expected output; if the intent is to further edit the result then we'll need the question updated to show the ultimate expected output

userrC93 · Answer

I would like to know if this could be achieved using only sed and its buffer.

The difficult part, in my opinion, is how can we increment the ID using sed? If you need to perform arithmetic, the usual choice would be to use AWK to get what you want. That said, you could (for better or worse) use GNU sed by piping input from a shell command into the pattern space. This uses the 'e' substitution flag, which is a GNU sed extension.

This solution copies the pattern space to hold space, appends "_old" and prints the pattern space. We can then copy the hold space to pattern space, capturing the ID and the remainder of the line. Our replacement string uses double parentheses to perform the arithmetic expansion (by adding six to the first capture group) and a printf statement to (1) pad the ID with up to two leading zeros and (2) insert the second capture group (i.e. the remaining part of the line):

sed '/hdfs/ { h; s/$/_old/p; g; s/^ID_$[^:]$*$.*$/printf "ID_%02d\2" $((\1+6))/e }' file

Results:

ID_00:/hdfs_01              name1_old
ID_06:/hdfs_01              name1

ID_01:/hdfs_02              name2_old
ID_07:/hdfs_02              name2

ID_02:/hdfs_03              name3_old
ID_08:/hdfs_03              name3

ID_03:/hdfs_app_data_01     name4_old
ID_09:/hdfs_app_data_01     name4

ID_04:/hdfs_app_data_02     name5_old
ID_10:/hdfs_app_data_02     name5

ID_05:/hdfs_cmmd_prt        name6_old
ID_11:/hdfs_cmmd_prt        name6

RavinderSingh13 · Answer

With your shown samples please try following awk code. Written and tested in GNU awk. Using its match function's capability of using regex to create capturing groups which are stores as elements in array to be used later on for requirement.

awk -v value=$(tail -1 Input_file | awk -F'[_:]' '{print $2}') '
NF{
  print $0"_old"
  match($0,/^(ID_)[^:]*(:.*)/,arr)
  print arr[1] sprintf("%02d",++value) arr[2]
  next
}
1
'   Input_file

Daweo · Answer

But I would like to know if this could be achieved using only sed and its buffer. Saving the original matched pattern, modify it once for the old suffix and print it [this is output #1] and then modify the original line again for the ID and print it again [this is output #2 ], then we can remove the original line.

If you are using GNU sed you then have commands which allows to so, consider simple example, you have file.txt with following content

a123b
c456d
e789f

and want to create fileold.txt and fileID.txt, 1st holding number followed by old and 2nd followed by ID that is

123old
456old
789old

and

123ID
456ID
789ID

then you could achieve this as follows

sed -e 's/[^0-9]*$[0-9]*$[^0-9]*/\1/' -e 'h' -e 's/$/old/' -e 'w fileold.txt' -e 'g' -e 's/$/ID/' -e 'w fileID.txt' file.txt

Explanation: I firstly modify pattern space, so it is just number using substitue with capturing group, after doing it I store it in hold space (h), then I change pattern space by replacing end of line using old, that is I append old, then I write pattern space to fileold.txt, then I ram content of hold space into pattern space, so it is back just number, then I append ID to and write that to fileID.txt

(tested in GNU sed 4.5)

potong · Answer

This might work for you (GNU sed and shell):

sed -E 'H;$!d;g
        s/ID.*/&_old
&/mg
        h;s/.*ID_(.*):.*/\1/;H;x
        :a;s/(ID_([^:+]*):.*ID_)\2(.*
)(.*)/\1$((1+\4))\31+\4/;ta
        s/.(.*)
.*/echo "\1"/e
        s/_(.):/_0\1:/mg' file

Slurp the whole file into the hold space.

Duplicate the lines containing ID's and append _old to the first line of each.

Make a copy and then extract the last ID number and append it.

Using substitution and a loop, iterate through the ids replacing the second id number by an incremented last id number and wrap the result in a shell arithmetic variable.

Ditch the first newline and the last id number and evaluate the result using shell to interpolate the shell variables instantiated in the previous step.

Touch up the final result by placing a leading zero for those ids that are less than 10.

Using SED to modify the same string twice and delete it

6 Answers