How to log the memory consumption on Linux?

I am a big fan of logging everything and I find it useful to know which processes are using the memory and how much each process is using (as well as sumary statistics). The following command records a top printout ordered by memory consumption every 0.5 seconds:

top -bd0.5 -o +%MEM > memory.log

Just note that the log file will grow a lot faster than if you only store the total memory utilization statistics so be sure you don't run out of disk space.

I think adding a crontab entry will be enough

*/5 *  *  *  *  free -m >> some_output_file

There are other tools like SeaLion, New Relic, Server Density etc which will almost do the same but are much easier to install and configure. My favorite is SeaLion, as it being free and also it gives a awesome timeline view of raw outputs of common linux commands.

To periodically log the memory usage efficiently, I combined another answer here with a method to only retain the top-K memory-using processes.

top -bd 1.5 -o +%MEM | grep "load average" -A 9 > memory_usage.log

This command will record, every 1.5s, the top header information and the 3 highest memory-consuming processes (there's a 6-line offset for top's header information). This saves lots of disk space over recording top's information for every process.

So I know that I am late to this game, but I just came up with this answer, as I needed to do this, and really didn't want the extra fields that vmstat, free, etc... all will seem to output without excess filtering. So here is the answer that I came up with:

top -bd 0.1 | grep 'KiB Mem' | cut -d' ' -f10 > memory.txt

OR:

top -bd 0.1 | grep 'KiB Mem' | cut -d' ' -f10 | tee memory.txt

the standard output from top when grep ing with Kib Mem is:

KiB Mem : 16047368 total,  8708172 free,  6015720 used,  1323476 buff/cache

By running this through cut, we filter down to literally just the number prior to used

The user can indeed modify the 0.1 to another number in order to run different capture sample rates. In my case I wanted to use top also because you can run memory stats faster than 1 second per capture, as you can see here I wanted to capture a stat every 1/10th of a second.

NOTES: It does turn out that piping through cut cause MASSIVE delay in getting anything out to file. As we later found out, it is much faster to leave out the cut command during data acquisition, then perform the cut command on the output file later. Also, we had no need for timestamps in our tests.

This thus looks as follows:

Begin Logging:

top -bd 0.1 | grep 'KiB Mem' | tee memory_raw.txt

Exit Logging:

ctrl-z (to exit logging)

Filter:

2 levels of cut (filtering), first by comma, then by space. This is due to the alignment of top and provides much cleaner output:

cut memory_raw -d',' -f3 | tee memory_used_withlabel.txt
cut memory_used_withlabel.txt -d' ' -f3 | tee memory_used.txt