Apache SshClient ServerKeyVerifier never gets invoked when i get an incoming connection

Im working in an Application that has a listener waiting for incoming connections. The listener gets the connection and authenticates the session. The problem is we want to identify the incoming connections by their publicKey and not the remoteAddress since the same device might change remoteAddress every time it calls but the sshKey would remain the same.

My problem is i cant seem to figure out how to read the public key from incoming connections.

Im using a java.nio AsynchronousServerSocketChannel as my listener. Application starts and listens for connections.

We get the asynchronousChannel from the incoming call and make a Nio2Session/ClientSession out of it from our own extended version of apache.sshd.Nio2Connector (CustomSessionChannelConverter)

We later use this sshClient to authenticate the clientSession. All of the above works as of now.

I added a serverKeyVerifier to the sshClient hoping it would be called during either the incoming call, or when we are trying to authenticate the session.

public CustomListener() throws IOException {
        factoryManager = SshClient.setUpDefaultClient();
        factoryManager.setKeyPairProvider(new SimpleGeneratorHostKeyProvider());

        KnownHostsServerKeyVerifier verifier = new DefaultKnownHostsServerKeyVerifier(new ServerKeyVerifier() {
            @Override
            public boolean verifyServerKey(ClientSession clientSession, SocketAddress remoteAddress, PublicKey serverKey) {

                //My goal is to check the public key here , but it never gets called
                return false;
            }
        }, false);

        factoryManager.setServerKeyVerifier(verifier);
        factoryManager.start();

        ExecutorService executorService = ((AbstractIoServiceFactory) (factoryManager.getIoServiceFactory())).getExecutorService();
        AsynchronousChannelGroup group = AsynchronousChannelGroup.withThreadPool(ThreadUtils.protectExecutorServiceShutdown(executorService, true));
        converter = new CustomSessionChannelConverter(factoryManager, factoryManager.getSessionFactory(), group);
        connection = new ConnectionHandler(converter);
    }

But the serverKeyVerifier never gets invoked. I dont know what im missing for the serverKeyVerifier to come into play. I'd appreciate any advice

I uploaded the 3 files to this dummy project in case that helps https://github.com/jorgen25/CustomListenerProject/tree/main/src/main/java

I used to have a test environment i can make calls from and listen on my localhost but its been unavailable for a few days and im trying to replicate the flow making ssh calls from my terminal if thats any help

Thanks again for your time!