I face a problem when running Microservices on K8s. Sometimes this error is shown
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
I was checking some config on the worker (all process running as root): ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 385975
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1048576
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
and cat /proc/sys/kernel/threads-max
771951
Memory is good
free -m
total used free shared buff/cache available
Mem: 96517 35835 45218 10 15462 60126
Swap: 0 0
Limits on each process is unlimited.
cat /proc/440583/limits | grep processes
Max processes unlimited processes
I created a script to counting the number of thread and write to file
while true; do ps -eo nlwp | tail -n +2 | awk '{ num_threads += $1 } END { print num_threads }' >> test.txt; date >> test.txt; sleep 10; done &
this is the log at the time of the error
Mon Jun 21 13:18:48 2021
34323
Mon Jun 21 13:18:52 2021
34325
Mon Jun 21 13:18:58 2021
34324
Mon Jun 21 13:19:02 2021
11945
Mon Jun 21 13:19:10 2021
11979
(the error occurs in 13:18)
All Microservices have good heap usage. It's seems ulimit or thread-max not working as expected. Could anyone help me?