Can I do a "for loop" with an awk '!seen[$0]' in there to apply in multiple .txt files?

Admin 1
Viewed 242

I want to delete some repeated lines in multiple .text files I need to analyze each file independently, since they are not related to each other using:

awk '!seen[$0]' file.txt file.out

But when I try: 

for f in *.txt; do awk 'seen![$0]' $f $f.out; done

I get an error:
The file '$f' cannot be opened. Awk or '! Seen [$ 0]' is not recognized Sometimes I get an output file but it's the same file ....

for-loop awk
1 Answers

With NON-GNU awk could you please try following.

awk -v temp_out="file.out" '
FNR==1{
  if(prev_filename){
    close(temp_out)
    sub(/\.txt/,".out",prev_filename)
    system("mv -- \047" prev_out "\047 \047" prev_filename "\047")
  }
  prev_filename=FILENAME
  delete seen
}
!seen[$0]++{
  print > (temp_out)
}
END{
  if(prev_filename){
    close(temp_out)
    sub(/\.txt/,".out",prev_filename)
    system("mv -- \047" prev_out "\047 \047" prev_filename "\047")
  }
}
' *.txt

Explanation: Adding explanation for above code.

awk -v temp_out="file.out" '                        ##Starting awk program from here with setting variable prev_out to file.out here.
FNR==1{                                             ##Checking condition if line is first line then do following.
  if(prev_filename){                                ##Checking if prev_filename is NOT NULL then do following.
    close(temp_out)                                 ##Closing prev_out file here from back-end.
    sub(/\.txt/,".out",prev_filename)               ##Substitute .txt with .out in previous filename here.
    system("mv -- \047" prev_out "\047 \047" prev_filename "\047")        ##Using system command to rename temp file prev_out with prev_filename(with .out)
  }
  prev_filename=FILENAME                            ##Setting prev_filename to current FILENAME here.
  delete seen                                       ##Deleting array seen here.
}
!seen[$0]++{                                        ##Checking if current line is NOT present in array seen then do following.
  print > (temp_out)                                ##Printing current line to temp file here.
}
END{                                                ##Starting END block of this program from here.
  if(prev_filename){                                ##Checking if prev_filename is NOT NULL then do following.
    close(temp_out)                                 ##Closing prev_out file here from back-end.
    sub(/\.txt/,".out",prev_filename)               ##Substitute .txt with .out in previous filename here.
    system("mv -- \047" prev_out "\047 \047" prev_filename "\047")        ##Using system command to rename temp file prev_out with prev_filename(with .out)
  }
}
' *.txt                                             ##Mentioning all .txt files here.
user_58665800
Related