We have two machines with identical configuration and use (we have two balanced Siebel application servers in them).
Normally, we have a very similar RAM usage in them (around 7 Gb). Recently, we've have a sudden increase of RAM in only one of them and now we have close to 14 Gb utilization of RAM in that machine.
So, for very similar boxes, we have one of them using 7Gb of RAM while the other one is consuming 14 Gb.
Now, using ps aux command to determine which process it's using all this additional memory, we see memory consumption is very similar in both machines. Somehow, we don't see any process that's using those 7 Gb of additional RAM.
Let's see:
Machine 1:
total used free shared buffers cached
Mem: 15943 15739 204 0 221 1267
-/+ buffers/cache: 14249 1693
Swap: 8191 0 8191
So, we have 14249 Mb usage of RAM.
Machine 2:
total used free shared buffers cached
Mem: 15943 15636 306 0 962 6409
-/+ buffers/cache: 8264 7678
Swap: 8191 0 8191
So, we have 8264 Mb usage of RAM.
I guess, the sum of Resident Set Size memory of ps should be equal or bigger to this value. According to this answer is how much memory is allocated to the process and is in RAM (including memory from shared libraries). We don't have any memory in SWAP.
However:
Machine 1:
ps aux | awk 'BEGIN {sum=0} {sum +=$6} END {print sum/1024}'
8357.08
8357.08 < 14249 -> NOK!
Machine 2:
ps aux | awk 'BEGIN {sum=0} {sum +=$6} END {print sum/1024}'
8468.63
8468.63 > 8264 -> OK
What do I get wrong? How can I find where this "missing" memory is?
Thank you in advance