Too many open files Exception in ECS Cloudwatch logs for a Java Service calling Lambda

I have a Java service that invokes a lambda to get stuff done. When the service keeps running for a while, it seems to throw the below shown exception. However I do not open files in the flow of the code. My service is running in an ECS container in AWS.

java.io.IOException: Too many open files
2021-11-25T11:17:30.850-08:00   at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:1.8.0_302]
2021-11-25T11:17:30.850-08:00   at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421) ~[?:1.8.0_302]
2021-11-25T11:17:30.850-08:00   at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249) ~[?:1.8.0_302]
2021-11-25T11:17:30.850-08:00   at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:540) ~[tomcat-coyote.jar:9.0.52]
2021-11-25T11:17:30.850-08:00   at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:78) ~[tomcat-coyote.jar:9.0.52]
2021-11-25T11:17:30.850-08:00   at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:106) ~[tomcat-coyote.jar:9.0.52]
2021-11-25T11:17:30.850-08:00   at java.lang.Thread.run(Thread.java:749) [?:1.8.0_302]
2021-11-25T11:17:31.412-08:00   java.io.FileNotFoundException: /proc/stat (Too many open files)

I tried increasing the ulimit of the container but that didn't help. I don't want to further increase it without understanding the root cause. One thing I noticed is that as this exception happens, number of open threads and file descriptor use are at their highest.

I have the below logic in the code, where I launch a few threads from ExecutorService.newFixedThreadPool of size 10, and invoke lambda in each of them using CompletableFutures. I'm doubting if this could be a problem.

public void createAdditionalTestDataForFuture(TestDataInput inputType, int existingTestDataCount)
            throws JsonProcessingException {
        for (int i = 0; i < additionalTestDataAmount - existingTestDataCount; i++) {
            String payload = this.prepareAdditionalTestData(inputType);
            CompletableFuture.supplyAsync(() -> persistenceManager.invokeLambda(payload), executorService)
                    .thenAccept(this::indexCreatedTestData);
        }
    }

The above method is called when a request reaches my service.

public O searchAndCreateTestDataIfNotFound(I inputType) throws IOException {
        Map<String, String> searchDocumentResults = persistenceManager.searchData(inputType).searchResults();
        Map<String, String> cleanSearchResults = this.getSearchResultsWithoutOverUsedData(searchDocumentResults);
        int existingValidTestDataCount = cleanSearchResults.size();

        if (existingValidTestDataCount > additionalTestDataAmount) {
            return getDocumentToReturnAndUpdateUsage(cleanSearchResults);
        } else if (existingValidTestDataCount == 0) {
            testDataGenerator
                    .createAdditionalTestDataForFuture(inputType, EXISTING_TEST_DATA_COUNT_FOR_LAMBDA_OUTPUT);
            return persistenceManager
                    .invokeLambdaAndIndexData(inputType);
        } else {
            testDataGenerator.createAdditionalTestDataForFuture(inputType, existingValidTestDataCount);
            return getDocumentToReturnAndUpdateUsage(cleanSearchResults);
        }
    }

Below is how I invoke my lambda.

public Map<String, String> invokeLambda(final String payload) {

        final InvokeRequest invokeRequest = new InvokeRequest()
                .withFunctionName(dataGeneratorAttributes.getLambdaArn())
                .withPayload(payload);

            final InvokeResult invokeResult = awsLambda.invoke(invokeRequest);

            if (!StringUtils.isEmpty(invokeResult.getFunctionError())) {
                throw new RuntimeException(invokeResult.getFunctionError());
            }

            final String output = new String(invokeResult.getPayload().array(), StandardCharsets.UTF_8);

            return GSON.fromJson(output, HashMap.class);

    }

Is there something wrong about the way I have handled the completable futures or invoking the lambda that could cause these too many open files? any ideas or suggestions to go about investigating this would be greatly appreciated. Thanks in advance.