I have a very large database (~ 50 million entries) built with latest ArangoDB Community Edition 3.7.8 and running in a proper (maybe even over) sized AWS EC2 instance. Most of the time memory usage is around 50% and CPU usage is average 20%.
The AWS EC2 instance is a:
c5.9xlarge
x86_64
vCPUs: 36
Memory: 72GB
Sometimes (once per day in random times) my backend java system stops accepting API requests and when I go to check the instance's CPU usage it's stuck 99% (used by arangodb, the java application remains with low CPU usage)
Now for example it's running fine
top - 19:31:50 up 23:49, 1 user, load average: 9.50, 10.72, 12.74
Tasks: 341 total, 1 running, 340 sleeping, 0 stopped, 0 zombie
%Cpu(s): 20.3 us, 0.5 sy, 0.0 ni, 79.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 48.0/70240.8 [|||||||||||||||||||||||||||||||||||||||||||||||| ]
MiB Swap: 0.0/0.0 [ ]
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
907 arangodb 20 0 103.5g 23.3g 43900 S 19.1 34.0 23096:40 arangod
36171 root 20 0 42.9g 7.3g 29908 S 0.7 10.7 360:44.63 java
When the problem starts it will be something like this (99.9% CPU usage)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
907 arangodb 20 0 103.5g 23.3g 43900 S 99.9 34.0 23096:40 arangod
36171 root 20 0 42.9g 7.3g 29908 S 0.7 10.7 360:44.63 java
I though it could be some problematic query that was being executed, but when the problem last occurred I checked the running and slow queries tabs and there was nothing wrong there... just fast queries being executed as always that takes up to 1 sec maximum. Some of them were a little slower, taking up to 3 secs, but that's all.
The database engine seemed locked doing something else.
My question is, how do I properly debug the database/engine to find out what's wrong?
Is there a command in arangosh to see which threads are being executed causing the system overload?
top - 16:04:29 up 8:30, 1 user, load average: 35.50, 29.85, 26.79
Threads: 5874 total, 39 running, 5835 sleeping, 0 stopped, 0 zombie
%Cpu(s): 87.0 us, 12.4 sy, 0.0 ni, 0.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 44.1/70240.8 [|||||||||||||||||||||||||||||||||||||||||||| ]
MiB Swap: 0.0/0.0 [ ]
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
34812 arangodb 20 0 68.2g 21.6g 38104 R 2.8 31.5 4:11.29 SchedWorker
34815 arangodb 20 0 68.2g 21.6g 38104 R 2.8 31.5 4:00.48 SchedWorker
34816 arangodb 20 0 68.2g 21.6g 38104 R 2.8 31.5 3:56.96 SchedWorker
34796 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 4:29.74 SchedWorker
34801 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 4:26.99 SchedWorker
34804 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 4:22.92 SchedWorker
34806 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 4:19.99 SchedWorker
34810 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 4:14.07 SchedWorker
34813 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 4:03.91 SchedWorker
34817 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 3:48.18 SchedWorker
34825 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 3:22.19 SchedWorker
34830 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 3:13.76 SchedWorker
34833 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 3:12.04 SchedWorker
35107 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 1:11.55 SchedWorker
34795 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 5:13.87 SchedWorker
34803 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 3:42.99 SchedWorker
34807 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 4:18.45 SchedWorker
34814 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 4:03.00 SchedWorker
34822 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 3:33.07 SchedWorker
34823 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 3:30.12 SchedWorker
34824 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 3:27.81 SchedWorker
34971 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 2:27.33 SchedWorker
34973 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 2:55.89 SchedWorker
34802 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 4:24.70 SchedWorker
34811 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 3:38.33 SchedWorker
34820 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 3:41.70 SchedWorker
34831 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 3:10.63 SchedWorker
34808 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 4:16.61 SchedWorker
34809 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 4:15.71 SchedWorker
34829 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 3:14.97 SchedWorker
35097 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 0:52.38 SchedWorker
35118 arangodb 20 0 68.2g 21.6g 38104 R 2.7 31.5 1:05.96 SchedWorker
34832 arangodb 20 0 68.2g 21.6g 38104 R 2.6 31.5 3:13.72 SchedWorker
34821 arangodb 20 0 68.2g 21.6g 38104 R 2.6 31.5 3:37.70 SchedWorker
34972 arangodb 20 0 68.2g 21.6g 38104 R 2.6 31.5 2:54.47 SchedWorker
34805 arangodb 20 0 68.2g 21.6g 38104 R 2.5 31.5 4:20.85 SchedWorker
35563 ubuntu 20 0 17712 10552 3196 R 0.3 0.0 0:00.74 top
904 arangodb 20 0 68.2g 21.6g 38104 S 0.1 31.5 1:36.13 jemalloc_bg_thd
997 arangodb 20 0 68.2g 21.6g 38104 S 0.1 31.5 2:00.13 jemalloc_bg_thd
1001 arangodb 20 0 68.2g 21.6g 38104 R 0.1 31.5 2:01.31 jemalloc_bg_thd
35572 root 20 0 43.1g 6.5g 29660 S 0.1 9.5 0:00.09 C2 CompilerThre
1923 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 0:29.03 https-jsse-nio-
1012 arangodb 20 0 68.2g 21.6g 38104 S 0.0 31.5 1:46.39 jemalloc_bg_thd
11 root 20 0 0 0 0 I 0.0 0.0 0:46.16 rcu_sched
1330 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 2:48.66 GC Thread#22
1593 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 0:28.92 https-jsse-nio-
1598 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 0:41.58 https-jsse-nio-
1832 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 0:28.95 https-jsse-nio-
1905 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 0:28.07 https-jsse-nio-
2018 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 0:28.44 https-jsse-nio-
2021 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 0:28.19 https-jsse-nio-
2026 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 0:28.55 https-jsse-nio-
2064 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 0:28.83 https-jsse-nio-
2082 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 0:28.11 https-jsse-nio-
2086 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 0:28.25 https-jsse-nio-
2117 root 20 0 43.1g 6.5g 29660 S 0.0 9.5 0:28.52 https-jsse-nio-
4900 root 20 0 43.1g 6.5g 29660 R 0.0 9.5 0:28.00 https-jsse-nio-
Thanks!