Input from an array for awk to find duplicates

user_8691119 0
Viewed 89

I tried to input data for awk from an array:

awk -v var="${A[*]}" 'BEGIN{split(var,list,"\n"); for (i=1;i<=length(list);i++) print list[i]}'

Also using awk to find duplicates between files:

filecnt=$(find "${pmdir}" -type f)
awk -v  n=filecnt '{a[$0]++}END{for (i in a)if (a[i]>1){print i, a[i];}}' $filecnt  >> ${outputfile}

But I had hard time to find out how to do it if awk takes an array as its input. something like:

awk -v var="${A[*]}" '{var[$0]++}END{for (i in var)if (var[i]>1){print i, var[i];}}'

A is a column data reading from a file:

for i in $( awk  -F ',' '{ print $1; }' "${ifile}" )
do
    A[$j]=$i
    #echo "${A[$j]}" 
    j=$((j+1))
done

example of A is

0x10000
0x11000
0x01100
0x00010
0x11000
0x00010
0x00010

The output is wanted:

0x11000 2
0x00010 3

Thanks for your suggestions.

sh awk shell
1 Answers

Is this what you want?

$ printf '%s\n' "${A[@]}" | sort | uniq -cd | awk '{print $2, $1}'
0x00010 3
0x11000 2

or if you prefer:

$ printf '%s\n' "${A[@]}" | awk '{cnt[$0]++} END{for (val in cnt) if (cnt[val]>1) print val, cnt[val]}'
0x11000 2
0x00010 3

or:

$ awk -v vals="${A[*]}" 'BEGIN{split(vals,tmp); for (i in tmp) cnt[tmp[i]]++; for (val in cnt) if (cnt[val]>1) print val, cnt[val]}'
0x11000 2
0x00010 3

Note that that last one relies on none of the values in A[] containing spaces or escape chars.

Your for loop isn't how to populate A[] in the first place, though, this is:

A=()
while IFS= read -r i; do
    A+=( "$i" )
done < <(cut -d',' -f1 "$ifile")

or:

A=()
while IFS=',' read -r i _; do
    A+=( "$i" )
done < "$ifile"

or:

readarray -t A < <(cut -d',' -f1 "$ifile")
user_17450010
Related