awk using filter less than 3 unique characters in column

Your code does something quite different than what you seem to be stating in your requirements. You loop over the tokens on the line (on the first line, there are two tokens, 1 and AAA) and check how many times you have seen those. Your script then prints if you have more than three unique tokens on the line, which you never do.

The concept you seem to be looking for is the substr() function, which allows you to loop over the individual characters in a string (for example).

awk 'BEGIN{FS="\t"; OFS="\t"}
{ unq=0; delete seen; for (i=1; i<=length($2); i++) if(!seen[substr($2, i, 1)]++) unq++ }
unq >= 3' file

To start debugging your own code, try adding a print statement at various strategic places to verify that you see what you expect; when you don't, you have found a bug. For example, it should be easy to see that for (i=1; i<=$NF; ++i) print $i loops over the fields of the input line.

With an awk that splits into chars given a null field separator and supports length(array) (e.g. GNU awk but also several others):

$ cat tst.awk
{
    delete chars
    split($2,tmp,"")
    for (i in tmp) {
        chars[tmp[i]]
    }
}
length(chars) >= 3

$ awk -f tst.awk file
1   AAABC
2   ABCD
2   ABCDE
3   AABCD
3   ABCDE
3   ABBCCDE

With perl:

$ perl -MList::Util=uniq -lane 'print if scalar(uniq split(//,$F[1])) > 2' ip.txt
1   AAABC
2   ABCD
2   ABCDE
3   AABCD
3   ABCDE
3   ABBCCDE

$F[1] has the second column, which is then split into characters using an empty regexp. From this array, duplicates are removed using uniq and then size of the array is obtained using scalar

Similar solution with ruby:

ruby -lane 'print if $F[1].chars.uniq.size > 2'

With your shown samples, please try following. Written and tested in GNU awk.

awk '
{
  count=0
  delete value
  num=split($2,arr,"")
  for(i=1;i<=num;i++){
    if(!value[arr[i]]++){ count++ }
  }
}
count>=3
' Input_file

Explanation: Adding detailed explanation for above.

awk '                                  ##Starting awk program from here.
{
  count=0                              ##Setting count to 0 here.
  delete value                         ##Deleting value array here.
  num=split($2,arr,"")                 ##Splitting 2nd field into arr array with delimiter of NULL.
  for(i=1;i<=num;i++){                 ##Traversing through all values of arr array here.
    if(!value[arr[i]]++){ count++ }    ##Checking if any element is NOT repeated in value then increase count by 1 here.
  }
}
count>=3                               ##Checking if count>=3 then print that line.
' Input_file                           ##Mentioning Input_file name here.

another approach

$ awk '{c=0; for(f=$2; f&&c<3; c++) gsub(substr(f,1,1),"",f)} c>2' file

1   AAABC
2   ABCD
2   ABCDE
3   AABCD
3   ABCDE
3   ABBCCDE