Is there any way to force make -j to not over consume my RAM? I work on a dev team, and we have different hardware sets, so -j8 may not be optimal for everyone. However, make -j uses too much RAM for me, and spills over into swap, which can take down my entire system. How can I avoid this?
Ideally, I would want make to watch the system load and stop spawning new threads, wait for some to complete, and continue on.
It is possible to limit process RAM usage using ulimit. But it may lead to failing process when limit will be exceeded. gcc loves to exceed any limit when linking in single thread. So ulimit solution is not popular.
Another solution is to provide estimate of gcc RAM usage per thread and maintain swap that will be used rarely. You can add load-average to stop spawning of new threads/jobs when load is too high.
I am using the following script /etc/profile.d/makeopts.sh (gentoo):
#!/bin/bash
# We need up to 1000 MB (less than 1GB) per thread.
MAX_THREADS=$(($(getconf _PHYS_PAGES) * $(getconf PAGE_SIZE) / (1000 ** 3)))
EFFECTIVE_THREADS=$(getconf _NPROCESSORS_ONLN)
THREADS=$((MAX_THREADS < EFFECTIVE_THREADS ? MAX_THREADS : EFFECTIVE_THREADS))
MAX_JOBS=$((MAX_THREADS / THREADS))
JOBS=$((MAX_JOBS < EFFECTIVE_THREADS ? MAX_JOBS : EFFECTIVE_THREADS))
MAX_LOAD=$((EFFECTIVE_THREADS * 9 / 10))
export MAKEOPTS="--jobs=$THREADS --load-average=$MAX_LOAD"
export EMERGE_DEFAULT_OPTS="--jobs=$JOBS --load-average=$MAX_LOAD"
Machine with 4 threads and 16 GB RAM:
MAKEOPTS="--jobs=4 --load-average=3"
EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=3"
Machine with 16 threads and 32 GB RAM:
MAKEOPTS="--jobs=16 --load-average=14"
EMERGE_DEFAULT_OPTS="--jobs=2 --load-average=14"
Please be aware that this config was created for CFLAGS="-O2 -pipe -march=native". Please re-estimate it if you want to add/remove light/heavy options.