awk create a list for testing elements

user_1303577 0
Viewed 577

I have a list of discrete elements that I want to test inclusion of an entry from each line of my file. I'd like a succinct way to create a list or array in awk and then test each line against that list.

My list of discrete elements:

ports=(1010, 2020, 3030, 8888, 12345)

myFile:

127.0.0.1 1010
127.0.0.1 1011
127.0.0.1 12345
127.0.0.1 3333

My pseudocode:

awk '
  BEGIN {
    test_ports=[1010, 2020, 3030, 8888, 12345]
  }
  ($2 in test_ports) {
    print $0
  }
' myFile

The code below works, but it is not succinct and I don't like how it grows as the list grows, like if I get 100 ports to test against, or 1000...

awk '
  BEGIN {
    test_ports["1010"]=1
    test_ports["2020"]=1
    test_ports["3030"]=1
    test_ports["8888"]=1
    test_ports["12345"]=1
  }
  ($2 in test_ports) {
    print $0
  }
' myFile

Something like this would be good too, but the syntax isn't quite right:

for i in 1010 2020 3030 8888 12345 {test_ports[i]=1}

EDIT

This code works too and is very close to what I need, but it still seems a bit long for what it's doing.

awk '
  BEGIN {
    ports="1010,2020,3030,8888,12345"
    split(ports, ports_array, ",")
    for (i in ports_array) {test_ports[ports_array[i]] = 1}
  }
  ($2 in test_ports) {
    print $0
  }
' myFile
awk
4 Answers

You may use it like this:

awk '
  BEGIN {
    ports = "1010 2020 3030 8888 12345"  # ports string
    split(ports, temp)                   # split by space in array temp 
    for (i in temp)                      # populate array test_ports
       test_ports[temp[i]]
  }

  $2 in test_ports                       # print rows with matching ports
' myFile

127.0.0.1 1010
127.0.0.1 12345

A note of explanation:

  • temp is a numerically indexed array where the ports (1010, 2020, etc) are the array values, indexed from 1
  • test_ports is an associative array where the ports are the array keys and the values are null.
  • the elem in array operator tests if the given element is an index (aka "subscript") of the array.

Addendum: You also have option of reading ports from a file if your ports list is big like this:

awk 'NR == FNR {ports[$1]; next} $2 in ports' ports.list myfile

Or else if you have ports saved in a string then use:

ports='1010 2020 3030 8888 12345'
awk 'NR==FNR{ports[$1]; next} $2 in ports' <(printf '%s\n' $ports) myfile

127.0.0.1 1010
127.0.0.1 12345
user_5482250

Since you said I'd like a succinct way to create a list or array in awk and then test each line against that list, here is a succinct way to create a list in awk and then test each line against that list:

$ awk 'index(",1010,2020,3030,8888,12345,",","$2",")' file
127.0.0.1 1010
127.0.0.1 12345

or if you prefer:

$ awk -v ports='1010,2020,3030,8888,12345' 'index(","ports",",","$2",")' file
127.0.0.1 1010
127.0.0.1 12345
user_17450010

Assuming you have a large number of ports (test strings) to test against, I'd suggest matching using two files instead of a string.

Let ports.txt be the file of ports and test.txt your input test file. Be ports.txt something like this:

1010
2020
3030
8888
12345

then run

awk 'NR==FNR{port[$0]=$0} ($2 in a){print}' ports.txt test.txt

this will create the port[] array from the first file and use it to print if matching in the second file.

This solution expands on the concept proposed in anubhava's answer, but with a concise syntax as you were looking for.

More info on the NR==NFR syntax here. A final note on re-usability: attached to an external process, you might have the same awk syntax running against the same test.txt file, changing the ports.txt file (e.g. ports1.txt, ports2.txt, portn.txt ...) so that you may match port groups instead.

user_21251100

Assuming you have the ports in ports.txt, then you might be able to use join:

$ cat ports.txt
1010
2020
3030
8888
12345
$ join -12 -o1.1,2.1 <(sort -bk2 myFile.txt) <(sort -b ports.txt)
127.0.0.1 1010
127.0.0.1 12345
user_8875390
Related