How to health check elasticsearch cluster from outside

Viewed 1173

I want to write a script to health check our elasticsearch cluster (deploy on kubernetes)

  1. I go inside pod which run elasticsearch master container and run below commands:
[elasticsearch@elasticsearch-master-0 ~]$ curl localhost:9200/frontend-dev-2021.12.03/_count
{"count":76,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}

[elasticsearch@elasticsearch-master-0 ~]$ curl localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 617,
  "active_shards" : 1234,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

As you can see, both index count and health check command are success. But when I run these command from outside (I give elasticsearch cluster an public endpoint)

root@ip-192-168-1-1:~# curl --user username:password esdev.example.com/frontend-dev-2021.12.03/_count
{"count":76,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}

root@ip-192-168-1-1:~# curl --user username:password esdev.example.com/_cluster/health
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

Only the index count command is success, the health check command always produce 403 Forbidden error.

I have searched and read through the official docs from elasticsearch but event the offcial docs only run command internal elasticsearch cluster or using kibana (http service kubernetes - internal k8s cluster).

How can I health check elasticsearch from outside? Or we can not do this because some mechanism of elasticsearch cluster?

Notes: I create a basic auth nginx (username:password) stand before the elasticsearch and this nginx has an ingressroute from traefik-v2

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    meta.helm.sh/release-name: basic-auth-nginx-dev
    meta.helm.sh/release-namespace: dev
  creationTimestamp: "2021-01-23T08:12:55Z"
  generation: 2
  labels:
    app: basic-auth-nginx-dev
    app.kubernetes.io/managed-by: Helm
  managedFields:
  - apiVersion: traefik.containo.us/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:meta.helm.sh/release-name: {}
          f:meta.helm.sh/release-namespace: {}
        f:labels:
          .: {}
          f:app: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        .: {}
        f:entryPoints: {}
        f:routes: {}
    manager: Go-http-client
    operation: Update
    time: "2021-01-23T08:12:55Z"
  name: basic-auth-nginx-dev-web
  namespace: dev
  resourceVersion: "103562796"
  selfLink: /apis/traefik.containo.us/v1alpha1/namespaces/dev/ingressroutes/basic-auth-nginx-dev-web
  uid: 5832b501-b2d7-4600-93b6-b3c72c420115
spec:
  entryPoints:
  - web
  routes:
  - kind: Rule
    match: Host(`esdev.example.com`) && PathPrefix(`/`)
    priority: 1
    services:
    - kind: Service
      name: basic-auth-nginx-dev
      port: 80
1 Answers

Could you please show us your nginx config?

  1. I think the problem come from your nginx because I see the output you show that nginx return 403 for you, not the elasticsearch.

  2. Could you please try another command start with _ like _template or something like that, there is a chance your nginx prevent access to path start with _ character.

Related