My JVM heap max is configured at 8GB on the name node for one of my hadoop clusters. When I monitor that JVM using JMX, the reported maximum is constantly fluctuating, as shown in the attached image.
http://highlycaffeinated.com/assets/images/heapmax.png
I only see this behavior on one (the most active) of my hadoop clusters. On the other clusters the reported maximum stays fixed at the configured value. Any ideas why the reported maximum would change?
Update:
The java version is "1.6.0_20"
The heap max value is set in hadoop-env.sh with the following line:
export HADOOP_NAMENODE_OPTS="-Xmx8G -Dcom.sun.management.jmxremote.port=8004 $JMX_SHARED_PROPS"
ps shows:
hadoop 27605 1 99 Jul30 ? 11-07:23:13 /usr/lib/jvm/jre/bin/java -Xmx1000m -Xmx8G
Update 2:
Added the -Xms8G switch to the startup command line last night:
export HADOOP_NAMENODE_OPTS="-Xms8G -Xmx8G -Dcom.sun.management.jmxremote.port=8004 $JMX_SHARED_PROPS"
As shown in the image below, the max value still varies, although the pattern seems to have changed.
http://highlycaffeinated.com/assets/images/heapmax2.png
Update 3:
Here's a new graph that also shows Non-Heap max, which stays constant:
