Fileset directory not excluding properly

Learning Ant - I'm trying to exclude all files under a certain directory with:

<fileset dir="${testclasses.dir}" id="test.files">
  <exclude name="**/directory-to-exclude/*.class"/>
</fileset>

and to make sure I excluded correctly, I'm printing out all the classes under that fileset like so:

<path id="updated.test.classes.path">
  <fileset dir="${testclasses.dir}"/>
</path>

<pathconvert pathsep="${line.separator}|   |-- "
             property="echo.path.compile"
             refid="updated.test.classes.path">
</pathconvert>
<echo message="Printing updated class path: ${echo.path.compile}"/>

Yet I still see:

     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass$$anon$1.class
     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass$$typecreator1$1.class
     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass$.class
     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass$Complex$.class
     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass$Complex.class
     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass$Deep$.class
     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass$Deep.class
     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass$LessDeep$.class
     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass$LessDeep.class
     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass$StrToTime.class
     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass$Struct$.class
     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass$Struct.class
     [echo] |   |-- /path/to/testfiles/directory-to-exclude/SomeTestClass.class

why is this happening?