We have a webapp that uses on average 20% CPU when idle, with no network traffic or any kind of requests.
It is running on Java 11, Tomcat 9, Spring Framework 5.3, Hibernate 5.4.
However the issues I will describe below were true on Java 8, Tomcat 8.5, Spring 4.3 and Hibernate 4. as well.
I tried to profile the application using JFR and JMC, and I experimented with a lot of configurations.
In the image above it looks like catalina-utility-1 and catalina-utility-2 threads wake up periodically and for a few seconds use a lot of CPU.
Also there seems to be a huge amount of memory allocations done by these threads, 30+ GB in total in the sampled 5 minutes interval.
For this profiling I've configured JFR to record everything at maximum, all options enabled.
When I tried to dig deeper into the details by looking at the Method Profiling details, I observed that it seems to be related to org.apache.catalina.webresources.Cache.getResource().
So I started to read about Tomcat caching and tried out different parameters to tune it via the context.xml file like this:
<Context>
<!-- Default set of monitored resources. If one of these changes, the -->
<!-- web application will be reloaded. -->
<WatchedResource>WEB-INF/web.xml</WatchedResource>
<WatchedResource>WEB-INF/tomcat-web.xml</WatchedResource>
<WatchedResource>${catalina.base}/conf/web.xml</WatchedResource>
<!-- Uncomment this to disable session persistence across Tomcat restarts -->
<!--
<Manager pathname="" />
-->
<Resources cachingAllowed="true" cacheMaxSize="3024000" cacheObjectMaxSize="10240" cacheTtl="10000"/>
</Context>
In this particular example, which is the one used for the JFR profiling, I increased cache size to 3GB and cacheTtl to 10 seconds. I thought that a larger cache and larger TTL would effect the interval of the CPU spikes because I suspected Tomcat was checking the cache (originally 1G in size) every 5 seconds which is the default.
However, whatever values I set for cache size or ttl, the periodical CPU spikes are identical.
And the cache size is big enough to hold whatever Tomcat wants to put in there because I increased the value after we saw warnings in the logs. Anyways, 1GB is enough to get rid of the warnings.
I also experimented with heap sizes ranging from 1 to 5GB, the profiling above was done with a 5GB heap size. I can't really go above this value without starting to hit physical memory limits.
We use G1GC as our garbage collector since the Java 8 days. Tuning its parameters did not effect CPU usage. I also tried out ParallelGC and SerialGC but the CPU usage pattern remained unchanged.
Searching Google for this kind of issues lead to no results and I am totally stuck on what else could I try or what else should I look at.
Any suggestions are welcome. Thanks.
UPDATE 1:
It seems like I had a formatting issue originally, and the opening <context> tag was missing from the context.xml when parsed. Fixed it.
I also tried, as suggested with <Context reloadable="false"> so that reloadable is explicitly set to false. It had absolutely no effect.
Is it possible to set the reloadable flag from anywhere else? I am speculating maybe some other file or setting applies it even if in context.xml it is set to false.
