Is there a way in python 3 to log the memory (ram) usage, while some program is running?
Some background info. I run simulations on a hpc cluster using slurm, where I have to reserve some memory before submitting a job. I know that my job require a lot of memory, but I am not sure how much. So I was wondering if there is a simple solution for logging the memory over time.
You can do that with the memory_profiler package. Just with adding a decorator @profile to a function, you will get an output like this:
Line # Mem usage Increment Line Contents
==============================================
3 @profile
4 5.97 MB 0.00 MB def my_func():
5 13.61 MB 7.64 MB a = [1] * (10 ** 6)
6 166.20 MB 152.59 MB b = [2] * (2 * 10 ** 7)
7 13.61 MB -152.59 MB del b
8 13.61 MB 0.00 MB return a
Otherwise, the easiest way to do it is to ask Slurm afterwards with the sacct -l -j <JobId> command (look for the MaxRSS column) so that you can adapt for further jobs.
Also, you can use the top command while running the program to get an idea of its memory consumption. Look for the RES column.
You can use subprocess module. Here is a sample output of bash command free
$ free -m
total used free shared buffers cached
Mem: 7979 7678 300 0 109 4628
-/+ buffers/cache: 2941 5038
Swap: 2046 360 1686
Python program -
import subprocess
result = subprocess.check_output(['bash','-c', 'free -m'])
free_memory = result.split('\n')[1].split()[3]
# print free_memory
# 300
If you want to check memory usage of some process or periodically log it, then you can use pmap or some other utility depending on your use case and then parse the output.