Trying to understand why my C++ program is slowing down after running for long time

user_15203755 0
Viewed 99

None of the threads close to the topic that I could find (like this one) helped me so far.

I have a C++ program that reads some data from a file, runs pretty complex and intensive operations and then writes outputs to new files. When I let the process run for few hours, here are the behaviours that I noticed:

  • The CPU usage is quite constant, but the RAM is slowly increasing until it reaches a sort of ceiling, then it becomes constant too.
  • The RAM usage 'ceiling' seems to increase depending on computer's RAM configuration. For example 3~4GB on a computer of 16GB of RAM, but 10~11GB on a computer of 64GB).
  • The average time to generate files of similar complexity increases over time and at some point becomes huge (giving the impression that the program is not running anymore).
  • When it starts slowing down that way, I kill the process and restart the generation of the same file (from scratch) with a fresh run of the program, and what took 3h before is finished in about 5min.

What can it be? I suspected memory leaking, but then wouldn't the RAM keep increasing indefinitely? What else can cause a program to progressively slow down, even when resources are available (CPU, RAM, disk space, etc.)?

c++ performance
1 Answers

I suspected memory leaking, but then wouldn't the RAM keep increasing indefinitely?

It is increasing indefinitely, you just don't see it in the RAM usage statistics that you are looking at.

What happens is that the OS will keep allocating new physical RAM pages up until the observed ceiling point. At that point whenever your program requests new memory it will swap out some of the already allocated memory to disk before allocating a new physical memory in RAM. Thereby you free physical RAM by swapping to disk at about the same rate as you are allocating virtual memory due to your leak. Since the swapped out memory probably only contains leaked data, it is unlikely that it will ever be touched again by your program, so it just keeps sitting on the disk indefinitely. If you let the program run long enough, eventually you will run out of swap space on disk as well.

The constant I/O to the disk caused by the swapping is likely the root cause for the observed slowdown.

Use a memory analysis tool like LeakSanitizer or Valgrind to detect the memory leak. Once your program's memory consumption stops growing constantly, the performance problems should go away.

user_5776030
Related