Yes, you read that right. I am trying to figure out a weird prod situation where we have a decent sized CouchDB instance (~75GB of data, 2000+ databases) deployed on a host with 31GB of memory. The issue I am running into is that under heavy load, Couch starts timing out and throwing errors like it has run out of system resources to use, but it is not maxing out on anything. CPU uses rises as expected but still only peaks around 70%.
It is the memory usage, though, that is very strange to me. Regardless of how much load is on the server, the memory usage remains essentially constant at ~2.5GB used and ~28GB in "buff/cache". I had expected the memory usage to be considerably higher (given there seems to be plenty of free memory on the server).
Has anyone seen this behavior before? It feels like there is some bottleneck that I am missing here that is preventing Couch from actually utilizing the available resources...
Also FWIW, here are some relevant performance settings from my Couch config:
[couchdb]
os_process_timeout = 60000
max_dbs_open = 5000