trying to get last 4 distinct values from a dynamic NF file

Viewed 115

I need to get the last 4 unique values from each record in a file. The file is delimited by comma and the number of columns can be different for different records. My input file.

$ cat last_cols.txt
F
F,B
F,B,A
F,B,A,F
F,B,A,F,B
F,B,A,F,B,G
F,B,A,F,B,G,E
F,B,A,F,B,G,E,F
F,B,A,F,B,G,E,F,E
F,B,A,F,B,G,E,F,E,B
F,B,A,F,B,G,E,F,E,B,A
F,B,A,F,B,G,E,F,E,B,A,D
F,B,A,F,B,G,E,F,E,B,A,D,F
F,B,A,F,B,G,E,F,E,B,A,D,F,E
F,B,A,F,B,G,E,F,E,B,A,D,F,E,E
F,B,A,F,B,G,E,F,E,B,A,D,F,E,E,D
$

My required output:

F,
B,F,
A,B,F,
A,B,F,
A,B,F,
A,B,F,G,
B,E,F,G,
B,E,F,G,
B,E,F,G,
B,E,F,G,
A,B,E,F,
A,B,D,E,
A,B,D,F,
A,D,E,F,
A,D,E,F,
A,D,E,F,

Attempt to get the output

 awk -F, ' { split("",arr); for(i=NF;i>0;i--) { if(!$i in arr && length(arr)<4 ) arr[$i]=1 }; for(i in arr) printf("%s,",i); print "" } ' last_cols.txt

I just get empty lines.

Since this is going to be run in bash env, I welcome other solutions as well.

6 Answers

Would you please try the following:

perl -F, -alne '
    for $f (reverse @F) {
        $seen{$f}++;
        last if (keys %seen) >= 4;
    }
    print join(",", sort keys %seen);
    undef %seen;
' last_cols.txt

With GNU awk, could you please try following, written and tested with shown samples. Output will be in sorted form as per OP.

awk '
BEGIN{
  FS=OFS=","
}
{
  val=""
  delete arr
  delete arr2
  for(i=NF;i>0;i--){
    if(!arr[$i]++){
       val=(val?val OFS:"")$i
    }
    if(length(arr)==4){
       break
    }
  }
  num=split(val,arr2,",")
  asort(arr2)
  for(key in arr2){
    printf("%s,%s",arr2[key],key==length(arr2)?ORS:"")
  }
}' Input_file

Output will be as follows.

F,
B,F,
A,B,F,
A,B,F,
A,B,F,
A,B,F,G,
B,E,F,G,
B,E,F,G,
B,E,F,G,
B,E,F,G,
A,B,E,F,
A,B,D,E,
A,B,D,F,
A,D,E,F,
A,D,E,F,
A,D,E,F,

Explanation: Adding detailed explanation for above.

awk '                             ##Starting awk program from here.
BEGIN{                            ##Starting BEGIN section of this program from here.
  FS=OFS=","                      ##Setting field separator and output field separator as comma here.
}
{
  val=""                          ##Nullifying val here.
  delete arr                      ##Deleting arr array here.
  delete arr2                     ##Deleting arr2 array here.
  for(i=NF;i>0;i--){              ##Running for loop from last field to first field each line.
    if(!arr[$i]++){               ##Checking condition if current field is NOT present in arr then do following.
       val=(val?val OFS:"")$i     ##Creating val which has current field value and keep on concatenating in its value.
    }
    if(length(arr)==4){           ##Checking condition if length of arr is 4 then do following.
       break                      ##Using break to come out of the loop.
    }
  }
  num=split(val,arr2,",")         ##Splitting val into arr2 with comma as delimiter.
  asort(arr2)                     ##Using asort to sorting values into arr2.
  for(key in arr2){               ##Going through arr2 here.
    printf("%s,%s",arr2[key],key==length(arr2)?ORS:"")   ##Printing values of arr2 one by one and printing new line when its last value of arr2.
  }
}
'  Input_file                     ##Mentioning Input_file name here.

Check this

awk -F, '{
    delete arr;
    for(i=NF;i>0;i--) {
        arr[$i]=1
        if (length(arr)==4) {
            break;
        }
    };
    for(i in arr) {
        printf("%s,",i);
    }
    printf("\n");
} ' last_cols.txt

Use this Perl one-liner:

perl -F',' -lane 'my %seen; my @u = grep { !$seen{$_}++ } reverse @F; splice @u, 4; print map { "$_," } sort @u;' last_cols.txt

The Perl one-liner uses these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-n : Loop over the input one line at a time, assigning it to $_ by default.
-l : Strip the input line separator ("\n" on *NIX by default) before executing the code in-line, and append it when printing.
-a : Split $_ into array @F on whitespace or on the regex specified in -F option.
-F',' : Split into @F on comma, rather than on whitespace.

my @u = grep { !$seen{$_}++ } reverse @F; : Reverse the array of strings that was read from the input line. Select the unique elements, keeping the order.
splice @u, 4; : Keep no more than the first 4 elements.
print map { "$_," } sort @u; : Sort the resulting strings ASCIIbetically, append commas and print.

SEE ALSO:
perldoc perlrun: how to execute the Perl interpreter: command line switches

Here is a way to get this output in gnu awk:

awk -F, 'BEGIN {PROCINFO["sorted_in"]="@ind_str_asc"} {delete a; for (i=NF; length(a)<4 && i>0; --i) a[$i]; $0=""; for (i in a) $0 = $0 i FS} 1' file

F,
B,F,
A,B,F,
A,B,F,
A,B,F,
A,B,F,G,
B,E,F,G,
B,E,F,G,
B,E,F,G,
B,E,F,G,
A,B,E,F,
A,B,D,E,
A,B,D,F,
A,D,E,F,
A,D,E,F,
A,D,E,F,

To make it more readable:

awk -F, 'BEGIN {
   PROCINFO["sorted_in"] = "@ind_str_asc"   # to sort by key ascending order
}
{
   delete a                                 # delete array a
   for (i=NF; length(a)<4 && i>0; --i)      # loop from last column backwards
      a[$i]                                 # store in array a and keep it sorted
   $0 = ""                                  # reset $0 to ""
   for (i in a)                             # loop through array a
      $0 = $0 i FS                          # append j and FS to $0
} 1                                         # print each record
' file

I'm able to get the result by adding brackets to the key check. !$i in arr to !($i in arr)

awk -F, ' { 
split("",arr); 
for(i=NF;i>0;i--) { 
  if(!($i in arr) && length(arr)<4 ) arr[$i]=1 }; 
  for(i in arr) printf("%s,",i); print "" 
} ' last_cols.txt


F,
B,F,
A,B,F,
A,B,F,
A,B,F,
A,B,F,G,
B,E,F,G,
B,E,F,G,
B,E,F,G,
B,E,F,G,
A,B,E,F,
A,B,D,E,
A,B,D,F,
A,D,E,F,
A,D,E,F,
A,D,E,F,
Related