Spring boot threads are waiting org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter:invokeHandlerMethod:827

We have developed set of APIs using spring boot. When performance test was run and hitting more than 5000 calls/minute, avg response time started increasing. When we investigated with the help of AppDynamics, more than 2% transactions are having slow response time (more than 1.5 seconds). But the CPU usage is still under 20%.

And all of them are waiting exactly at the same location i. e org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter:invokeHandlerMethod:827.

Below is the complete call graph.

HTTPServlet:service:742 (method time = 0 ms, total time = 1800 ms)
HTTPServlet:service:661 (method time = 0 ms, total time = 1800 ms)
Servlet - dispatcherServlet:doService:897 (method time = 0 ms, total time = 1800 ms)
Servlet - dispatcherServlet:doDispatch:963 (method time = 0 ms, total time = 1800 ms)
Web Service - org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter:handleInternal:738 (method time = 0 ms, total time = 1800 ms)
Web Service - org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter:invokeHandlerMethod:827 (method time = 0 ms, total time = 1800 ms)
java.util.concurrent.locks.LockSupport:parkNanos:215 (method time = 1800 ms, total time = 1800 ms)

What is the root cause for threads handing at this location? Are there any SpringBooot properties to be updated to eliminate this?

Following is the source code of RequestMappingHandlerAdapter:invokeHandlerMethod and it hangs at invocableMethod.invokeAndHandle(webRequest, mavContainer);

protected ModelAndView invokeHandlerMethod(HttpServletRequest request, HttpServletResponse response, HandlerMethod handlerMethod) throws Exception {

    ServletWebRequest webRequest = new ServletWebRequest(request, response);
    try {
        WebDataBinderFactory binderFactory = getDataBinderFactory(handlerMethod);
        ModelFactory modelFactory = getModelFactory(handlerMethod, binderFactory);

        ServletInvocableHandlerMethod invocableMethod = createInvocableHandlerMethod(handlerMethod);
        invocableMethod.setHandlerMethodArgumentResolvers(this.argumentResolvers);
        invocableMethod.setHandlerMethodReturnValueHandlers(this.returnValueHandlers);
        invocableMethod.setDataBinderFactory(binderFactory);
        invocableMethod.setParameterNameDiscoverer(this.parameterNameDiscoverer);

        ModelAndViewContainer mavContainer = new ModelAndViewContainer();
        mavContainer.addAllAttributes(RequestContextUtils.getInputFlashMap(request));
        modelFactory.initModel(webRequest, mavContainer, invocableMethod);
        mavContainer.setIgnoreDefaultModelOnRedirect(this.ignoreDefaultModelOnRedirect);

        AsyncWebRequest asyncWebRequest = WebAsyncUtils.createAsyncWebRequest(request, response);
        asyncWebRequest.setTimeout(this.asyncRequestTimeout);

        WebAsyncManager asyncManager = WebAsyncUtils.getAsyncManager(request);
        asyncManager.setTaskExecutor(this.taskExecutor);
        asyncManager.setAsyncWebRequest(asyncWebRequest);
        asyncManager.registerCallableInterceptors(this.callableInterceptors);
        asyncManager.registerDeferredResultInterceptors(this.deferredResultInterceptors);

        if (asyncManager.hasConcurrentResult()) {
            Object result = asyncManager.getConcurrentResult();
            mavContainer = (ModelAndViewContainer) asyncManager.getConcurrentResultContext()[0];
            asyncManager.clearConcurrentResult();
            if (logger.isDebugEnabled()) {
                logger.debug("Found concurrent result value [" + result + "]");
            }
            invocableMethod = invocableMethod.wrapConcurrentResult(result);
        }

        invocableMethod.invokeAndHandle(webRequest, mavContainer);
        if (asyncManager.isConcurrentHandlingStarted()) {
            return null;
        }

        return getModelAndView(mavContainer, modelFactory, webRequest);
    }
    finally {
        webRequest.requestCompleted();
    }
}