I have an app to send and receive messages (simplified). With these routes when you send an HTTP request to http://0.0.0.0:8089/put-message a JMS message is then sent to my-queue, and then when an HTTP request is sent to http://0.0.0.0:8089/get-message Camel tries to consume a JMS message from my-queue and return the body of that message to the HTTP client:
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>
<bean id="artemisConnectionFactory" class="org.apache.activemq.artemis.jms.client.ActiveMQJMSConnectionFactory">
<argument index="0" value="${jms.url}"/>
<argument index="1" value="${jms.username}"/>
<argument index="2" value="${jms.password}"/>
</bean>
<bean id="pooledConnectionFactory" class="org.messaginghub.pooled.jms.JmsPoolConnectionFactory"
init-method="start" destroy-method="stop">
<property name="maxConnections" value="10"/>
<property name="maxSessionsPerConnection" value="500"/>
<property name="connectionFactory" ref="artemisConnectionFactory"/>
</bean>
<bean id="jms" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory" ref="pooledConnectionFactory"/>
</bean>
<camelContext xmlns="http://camel.apache.org/schema/blueprint">
<route>
<from uri="jetty:http://0.0.0.0:8089/put-message"/>
<inOnly uri="jms:my-queue"/>
</route>
<route >
<from uri="jetty:http://0.0.0.0:8089/get-message"/>
<pollEnrich timeout="0">
<simple>jms:queue:my-queue</simple>
</pollEnrich>
</route>
</camelContext>
</blueprint>
Stack:
- ActiveMQ Artemis 2.25
- Camel 2.23.2
- Karaf 4.2
- messaginghub/pooled-jms 1.0.6
- artemis-jms-client-osgi 2.7.0
I am doing load testing (Apache Bench 5kb messages, 100 threads for sending and 100 threads for receiving), and I observe the following. The service works for half an hour without problems (queue does not increase in size), but then the time for receiving messages from the queue increases. I was able to reduce some metrics and noticed that it increases oldGen (at the same time, the queue has not yet left in the PAGE mode).
Based on the recommendation in the ActiveMQ Artemis tuning documentation XX:+UseParallelOldGC was used when starting the broker, but even if you set another mode the behavior does not change.
Here is a screenshot of memory utilization from the JVM where the broker is running:
You can see how the time to receive messages from the queue increases. However, it's worth noting that if you use permanent consumers (there is no re-creation of connections aka <from uri="jms:queue:my-queue" />) then the work is stable.
What could be the reason for the increase memory oldgen?
What other metrics can I apply to deal with the problem?
UPD: heap dump (when performance loss)
UPD 2: and heap dump(when i use Performance Tools ./artemis perf client --message-size 5000 --persistent queue://TEST_QUEUE) - no leak
