I have a single Java application running on eJDK 1.8.201. I observed that the system reported RSS usage of the java process keeps increasing, it will reach over than 90% of the physical memory that is 512M, and my code will reboot the system. On the other hand, the usedMemory() I printed out periodically from the code shows that it always fluctuates within a limited range no more than 37% of the physical memory (means <= 190M). From my understanding, that means the JVM was not returning it's free memory to the OS. Is this understanding correct? I've set the -Xmx=256M -Xms=256M, so I think the -Xmx options was not working.
Do you have any idea about what was happening and how to fix the issue?
The system is an embedded Linux on an ARMv7 board.