Unable to start ElasticSearch Pod

user_11168925 0
Viewed 159

A few months back, I deployed the Elastic-Search (version - 8.0.1) on Kubernetes (GCP) as a service as External load balancer using this guide.

Now, I am unable to perform any read or write operation on ElasticSearch. I checked the logs, in which I found that memory of the node is almost full.

Here are some logs which support this analysis:

flood stage disk watermark [95%] exceeded on [hulk-es-default-0][/usr/share/elasticsearch/data] free: 18.5mb[1.8%], all indices on this node will be marked read-only

Cluster health status changed from [YELLOW] to [RED] (reason: [shards failed [[1][0]]]).

This node is unhealthy: health check failed on [/usr/share/elasticsearch/data].`

Here are the errors that are coming when performing any read/write operation:

elasticsearch.exceptions.TransportError: TransportError(503, 'master_not_discovered_exception', None)

elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPSConnectionPool(host='...', port=****): Read timed out. (read timeout=30))

I increased the capacity of my elasticsearch persistent volume claim(PVC) but was unable to create the pod with that new volume.

I followed the following steps -

  • Set the allowVolumeExpansion field to true in their StorageClass object(s)

  • Scaled ElasticSearch Operator Deployment to 0 Replicas.

  • Deleted the statefulset Object without deleting the pods using

    kubectl delete sts <statefulset-name> --cascade=orphan

Before deleting I saved the yaml of the statefulset using

kubectl get sts <statefulset-name> -o yaml
  • Increased the storage in capacity in the yaml file of PVC.
  • Recreated the StatefulSet with the new storage request by the yaml file I saved using

kubectl apply -f file-name.yml

  • Scaled back the operator deployment to 1

But, when I recreated the stateful set, the CrashLoopBackOff error is being shown every-time.

Following are some logs -

  • readiness probe failed
  • Likely root cause: java.io.IOException: No space left on device
  • using data paths, mounts [[/usr/share/elasticsearch/data (/dev/sdb)]], net usable_space [0b], net total_space [975.8mb], types [ext4]

The persistent disk's volume that the ES pod is accessing is increased but still the pod is unable to start. Can anyone guide me here, what is the problem here ?

elasticsearch kubernetes google-cloud-platform
2 Answers

Appears to be an issue with the disk space ( /usr/share/elasticsearch/data). Verify that the persistent volume with the required size is available and is bound to the elasticsearch pod through PVC object.

user_32707850

It seems, for some reason the Pod is not seeing the new volume Size. Have you tried this, for the GKE Volume expansion?

If the Pod is always in the CrashLoopBackOff state you can use e.g: kubectl debug mypod -it --image=busybox So you will attach a debug container to your Pod and check what is going on with the mounted volume.

Others things you can also do is to create a snapshot/backup of your Volume and restore it on a new bigger volume to see if the Issue still persist.

user_116130130
Related