Spring Integration Sftp : Sometimes taking over 15 min to complete the operation

I am using Spring integration sftp to put files on a remote server and below is configuration.

<spring-integration.version>5.2.5.RELEASE</spring-integration.version>

I have configurated @MessagingGateway.

@MessagingGateway
public interface SftpMessagingGateway {
 
     @Gateway(requestChannel = "sftpOutputChannel")
     void sendToFTP(Message<?> message);
     
}

I have configured the MessageHandler as below,

@Bean
@ServiceActivator(inputChannel = "sftpOutputChannel" )
public MessageHandler genericOutboundhandler() {
    SftpMessageHandler handler = new SftpMessageHandler(outboundTemplate(), FileExistsMode.APPEND);
    handler.setRemoteDirectoryExpressionString("headers['remote_directory']");
    handler.setFileNameGenerator((Message<?> message) -> ((String) message.getHeaders().get(Constant.FILE_NAME_KEY)));
    handler.setUseTemporaryFileName(false);
    return handler;
}

I have configured SftpRemoteFileTemplate as below

private SftpRemoteFileTemplate outboundTemplate;

public SftpRemoteFileTemplate outboundTemplate(){
    if (outboundTemplate == null) {
        outboundTemplate = new SftpRemoteFileTemplate(sftpSessionFactory());
    }
    return outboundTemplate;
}

This is the configuration for SessionFactory

public SessionFactory<LsEntry> sftpSessionFactory() {
    DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory();
    factory.setHost(host);
    factory.setPort(port);
    factory.setUser(username);
    factory.setPassword(password);
    factory.setAllowUnknownKeys(true);
    factory.setKnownHosts(host);
    factory.setSessionConfig(configPropertiesOutbound());
    CachingSessionFactory<LsEntry> csf = new CachingSessionFactory<LsEntry>(factory);
    csf.setSessionWaitTimeout(1000);
    csf.setPoolSize(10);
    csf.setTestSession(true);
    return csf;
}

I have configured all this in one of the service.

Now the problem is,

Sometimes the entire operation takes more than 15 min~ specially if the service is ideal for few hours and I am not sure what is causing this issue. It looks like it is spending time on getting the active session from CachedSessionFactory the after operations are pretty fast below is the snap from one of the tool where I have managed to capture the time.

It usually takes few miliseconds to transfer files.

I have recently made below changes but before that as well I was getting the same issue,

• I have set isShareSession to false earlier it was DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
• There was no pool size I have set it to 10

I think I have configured something incorrectly and that's why I end up piling connection ? Or there is something else ?

Observation :

The time taking to complete the operation is somewhat similar all the time when issue occurs i.e 938000 milliseconds +

If I restart the application daily it works perfectly fine.