About the problem

We are using the AWS Elasticsearch Service to host our elasticsearch cluster.

We have an external system self hosted on our own server, which is not in the cloud. This system is pushing new data into the cluster every few hours.

On a few occasions we are experiencing exceptions in this external system. We are using the official elasticsearch php client for communicating with the cluster. An Elasticsearch\Common\Exceptions\ServerErrorResponseException is thrown during an import. This only occurs about 5-7 times a week. On the same time, the read requests from our applications on ec2 instances are working fine. There are no exceptions.

We are using several bulk-requests to import the new data as efficient as possible. Each bulk request contains 1000 small documents which have only 6 fields. The longest text in each field is about 40 characters long.

The error

The exception on the external system:

{
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "blocked by: [SERVICE_UNAVAILABLE/2/no master];"
      }
    ],
    "type": "cluster_block_exception",
    "reason": "blocked by: [SERVICE_UNAVAILABLE/2/no master];"
  },
  "status": 503
}

First configuration

Our first elasticsearch domain configuration with 3 data nodes and without dedicated master nodes:

Elasticsearch Version: 5.1
Status: Green
Number of nodes: 3
Number of data nodes: 3
Active primary shards: 2
Active shards: 5
Relocating shards: 0
Initializing shards: 0
Unassigned shards: 0
Instance count: 3
Enable dedicated master: yes
Dedicated master instance count: 3
Enable zone awareness: no

EBS volume size per instance: 30 GB
Total cluster size: 90 GB
Used size: about 4.3 GB

Current configuration

AWS recommends to have dedicated master nodes. So we have adjusted our domain config like this to 3 data and 3 master nodes. This is our current configuration:

Elasticsearch Version: 5.1
Status: Green
Number of nodes: 6
Number of data nodes: 3
Active primary shards: 2
Active shards: 5
Relocating shards: 0
Initializing shards: 0
Unassigned shards: 0
Instance count: 3
Enable dedicated master: yes
Dedicated master instance count: 3
Enable zone awareness: no

EBS volume size per instance: 30 GB
Total cluster size: 90 GB
Used size: about 4.3 GB

But the exception is still sometimes thrown and the rate of the error did not change.

My thoughts

I've thought about enabling the zone awareness setting. Maybe there are problems in the availability zone? But this is only possible if we use an even number of instances, and I am not sure how the cluster works under this condition if a split-brain happens.

TLDR

Why does elasticsearch randomly tells me that it does not find a master node when I have three dedicated master nodes?