Tomcat's Catalina utility threads are periodically using high CPU and memory

Viewed 2191

We have a webapp that uses on average 20% CPU when idle, with no network traffic or any kind of requests. It is running on Java 11, Tomcat 9, Spring Framework 5.3, Hibernate 5.4. However the issues I will describe below were true on Java 8, Tomcat 8.5, Spring 4.3 and Hibernate 4. as well. I tried to profile the application using JFR and JMC, and I experimented with a lot of configurations. enter image description here In the image above it looks like catalina-utility-1 and catalina-utility-2 threads wake up periodically and for a few seconds use a lot of CPU. Also there seems to be a huge amount of memory allocations done by these threads, 30+ GB in total in the sampled 5 minutes interval.

For this profiling I've configured JFR to record everything at maximum, all options enabled.

enter image description here

When I tried to dig deeper into the details by looking at the Method Profiling details, I observed that it seems to be related to org.apache.catalina.webresources.Cache.getResource().

So I started to read about Tomcat caching and tried out different parameters to tune it via the context.xml file like this:

<Context>
  <!-- Default set of monitored resources. If one of these changes, the    -->
  <!-- web application will be reloaded.                                   -->
  <WatchedResource>WEB-INF/web.xml</WatchedResource>
  <WatchedResource>WEB-INF/tomcat-web.xml</WatchedResource>
  <WatchedResource>${catalina.base}/conf/web.xml</WatchedResource>
  <!-- Uncomment this to disable session persistence across Tomcat restarts -->
  <!--
    <Manager pathname="" />
    -->
  <Resources cachingAllowed="true" cacheMaxSize="3024000" cacheObjectMaxSize="10240" cacheTtl="10000"/>
</Context>

In this particular example, which is the one used for the JFR profiling, I increased cache size to 3GB and cacheTtl to 10 seconds. I thought that a larger cache and larger TTL would effect the interval of the CPU spikes because I suspected Tomcat was checking the cache (originally 1G in size) every 5 seconds which is the default. However, whatever values I set for cache size or ttl, the periodical CPU spikes are identical. And the cache size is big enough to hold whatever Tomcat wants to put in there because I increased the value after we saw warnings in the logs. Anyways, 1GB is enough to get rid of the warnings.

I also experimented with heap sizes ranging from 1 to 5GB, the profiling above was done with a 5GB heap size. I can't really go above this value without starting to hit physical memory limits.

We use G1GC as our garbage collector since the Java 8 days. Tuning its parameters did not effect CPU usage. I also tried out ParallelGC and SerialGC but the CPU usage pattern remained unchanged.

Searching Google for this kind of issues lead to no results and I am totally stuck on what else could I try or what else should I look at.

Any suggestions are welcome. Thanks.


UPDATE 1:

It seems like I had a formatting issue originally, and the opening <context> tag was missing from the context.xml when parsed. Fixed it.

I also tried, as suggested with <Context reloadable="false"> so that reloadable is explicitly set to false. It had absolutely no effect.

Is it possible to set the reloadable flag from anywhere else? I am speculating maybe some other file or setting applies it even if in context.xml it is set to false.

1 Answers

The stack trace in your images contains a call to Loader#modified and is only possible if you set the reloadable property of your context to true:

<Context reloadable="true">
...
</Context>

As described in Tomcat's documentation:

Set to true if you want Catalina to monitor classes in /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically reload the web application if a change is detected. This feature is very useful during application development, but it requires significant runtime overhead and is not recommended for use on deployed production applications. That's why the default setting for this attribute is false.

(emphasis mine).

Set reloadable to false (or delete the attribute) to get rid of the overhead.

Related