ElasticSearch's MasterService taking too long to compute cluster state and throw ProcessClusterEventTimeoutException

user_303914 0
Viewed 4071

We have an application that adds thousands of documents per second to our ES cluster. Each time we roll over the indices to which we are writing, and we start writing to new indices, we get the following errors, which don't allow ingesting documents for about 1 minute. After that 1 minute, everything goes back to normal until we rollover the index again.

[WARN ][o.e.c.s.MasterService    ] [NODE_NAME_1] took [28.3s], which is over [10s], to compute cluster state update for [put-mapping[_doc, _doc, ...
[DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [NODE_NAME_1] failed to put mappings on indices [[[INDEX_1/SOME_ID]]], type [_doc]
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
    at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$0(MasterService.java:143) [elasticsearch-7.5.2.jar:7.5.2]
    at java.util.ArrayList.forEach(ArrayList.java:1507) [?:?]
    at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:142) [elasticsearch-7.5.2.jar:7.5.2]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:830) [?:?]

In the first line, I put ellipsis at the end as it's actually quite large, here an image of what we're seeing (the line ends abruptly as you see in the image):

enter image description here

Do you know what those error messages are about?

Is it possible we are seeing those messages because of our cluster state being too large? Would that mean we have too many shards or nodes?

Thanks.

Note:

  • We use ElasticSearch 7.5.2
  • We don't use ILM, when I say "roll over our indices" what we do is to start writing to new indices that we created hours ago but were not using. Then we stop writing to the old indices.
elasticsearch
1 Answers

This error comes when ES is not able to cope up with the large cluster state, and the default timeout to update cluster state is 10s, but as mentioned in your error message, it took ~30 seconds.

Cluster state is computed on the master node and then publish a Diff(node was present) or entire cluster state(if a new node joins a cluster) by MasterService.java class of Elasticsearch.

you have huge no of shards information, which is maintained in the cluster state, your master node is not able to compute state within default 10s.

Below is the code from Elasticsearch class org.elasticsearch.cluster.service.ClusterApplierService, which tries to update the cluster state and throws this exception.

try {
    UpdateTask updateTask = new UpdateTask(config.priority(), source, new SafeClusterApplyListener(listener, logger), executor);
    if (config.timeout() != null) {
        threadPoolExecutor.execute(updateTask, config.timeout(),
            () -> threadPool.generic().execute(
                () -> listener.onFailure(source, new ProcessClusterEventTimeoutException(config.timeout(), source))));
    } else {
        threadPoolExecutor.execute(updateTask);
    }
}

And you can check the exception class org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException code as below

public ProcessClusterEventTimeoutException(TimeValue timeValue, String source) {
        super("failed to process cluster event (" + source + ") within " + timeValue);
    }

You can look at the above code, which will help you to better understand the problem.

Some suggestions to fix the issues:

  1. Increase the heap size and CPU of master node, so that the cluster state can be computed quickly as in your case, even the first step of calculating takes 30 sec. which means your master nodes doesn't have enough infrastructure to compute it fast.
  2. Delete unnecessary indices, which would reduce the no of shards and cluster state.
  3. Below are settings for publishing the cluster state, this is the second step, if you see publishing taking a long time, then you can do it if you are OK your some if data nodes become unavailable for this duration.
cluster.publish.info_timeout = 10s default(maybe this can work)
cluster.publish.timeout = 30s by default
user_40394310
Related