we have been using a 3 node Elasticsearch(7.6v) cluster running in docker container. I have been experiencing very high cpu usage on 2 nodes(97%) and moderate CPU load on the other node(55%). Hardware used are m5 xlarge servers. There are 5 indices with 6 shards and 1 replica. The update operations take around 10 seconds even for updating a single field. similar case is with delete. however querying is quite fast. Is this because of high CPU load? 2 out of 5 indices, continuously undergo a update and write operations as they listen from a kafka stream. size of the indices are 15GB, 2Gb and the rest are around 100MB.
1 Answers
You need to provide more information to find the root cause:
- All the ES nodes are running on different docker containers on the same host or different host?
- Do you have resource limit on your ES docker containers?
- How much heap size of ES and is it 50% of host machine RAM?
- Node which have high CPU, holds the 2 write heavy indices which you mentioned?
- what is the refresh interval of your indices which receives high indexing requests.
- what is the segment size of your 15 GB indices, use https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-segments.html to get this info.
What all you have debugged so far and is there is any interesting info you want to share to find the issue?
Related