In my C++ program running in linux (Ubuntu 14.4), I need to read a 90 GB file completely buffered in a C++ vector, and I have only 125 GB memory.
When I read the file chunk by chunk, it continuously results in increase in cached mem usage in linux, which turns out to be more than 50% of the 128 GB mem, then the free memory easily becomes under 50 GB.
total used free shared buff/cache available
Mem: 125 60 0 0 65 65
Swap: 255 0 255
So I found that the free memory then becomes zero, and the file reading process almost got stopped, and I have to manually run:
echo 3 | sudo tee /proc/sys/vm/drop_caches
to clear cached mem, so that the file reading process resumes. I understand cached mem is to speed up reading a file again. My question is how could I avoid manually running the drop cache command to ensure the file reading process would successfully complete?