Identifying a node's zone from inside a Kubernetes pod/container

user_10389992 0
Viewed 1430

I have a use case with elastic search for rack awareness, which requires me to identify the zone that a pod has been scheduled in.

I've seen many requests for this outside of SO, such as:

https://github.com/kubernetes/kubernetes/issues/40610

The zone is not exposed in the downward API due to it being a node label, not a pod label.

The two "answers" I have come up with are:

a) Curl request to the google metadata endpoint to retrieve the node from the compute engine metadata

b) Identify the node name via the downward API, and make a call to the Kube API with the node name to retrieve the node object, and use a tool such as JQ to filter the JSON response to get the zone.

I don't like option B due to it being more or less hardcoded against the API call, and I would need to provision a custom docker image with JQ and curl included. Option A feels a bit 'hacky' given it's not Kube native.

Are there any better solutions that I've missed?

Thanks in advance,

google-kubernetes-engine kubernetes
4 Answers

I don't particularly like doing it this way but I've yet to find a better answer, and none of the feature requests or bug reports on github seem to be going anywhere.

I opted to use a config map with a bash script which would do the curl request and some string manipulation, and then mount this into the container with volumes/volumeMounts and set a custom entry point to execute the script; injecting values into elasticsearch.yml, and then execute ES itself:

apiVersion: v1
kind: ConfigMap
metadata:
  name: elastic
data:
  elastic.sh: |-
    set -e
    elasticConfig="/usr/share/elasticsearch/config/elasticsearch.yml"

    #this gives the us the node zone in the format projects/{projectnumber}/zones/{zone}
    zone=$(curl -sS http://metadata.google.internal/computeMetadata/v1/instance/zone -H 'Metadata-Flavor: Google')

    #Split and retain the text after the last /
    zone=${zone##*/}

    #Append it to elasticsearch.yml

    echo "\nnode.attr.zone: ${zone}" >> $elasticConfig
    echo "\ncluster.routing.allocation.awareness.attributes: zone" >> $elasticConfig

    echo "\ncluster.routing.allocation.awareness.force.zone.values: {{ .Values.elastic.shardAwareness.zones }}" >> $elasticConfig

I don't particularly like this solution as it adds unnecessary overhead and isn't kube native. Querying the kube API is possible but has its own set of complications and hacks. I hope one day either the downward API will expose zone / region labels, or that maybe I'm missing something and there is a better way than this after all.

If nothing else, perhaps this will help someone else and stop them wasting time googling for answers that don't seem to be out there!

user_103899920

My case is Google Kubernetes Engine, and I needed to have zone/region labels on pods.

What i did was combined @dwillams782 approach in https://stackoverflow.com/a/52428782/8547463 of getting zone via 

curl -sS http://metadata.google.internal/computeMetadata/v1/instance/zone -H 'Metadata-Flavor: Google'`

and set it as labels to pods via approach in https://github.com/devth/k8s-labeler. Basically i called the zone detection API from initContainer of pod.

user_85474630
user_1472650

I think you can use pod anti-affinity to do this, here is a example:

spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: security
              operator: In
              values:
              - S2
          topologyKey: failure-domain.beta.kubernetes.io/zone

More detail refer to kubernetes docs

user_75276750
Related