Can anyone clarify where you can specify KUBERNETES_SERVICE_HOST in azure AKS for other namespaces? Kube-system is correct, other namespace not

user_13760942 0
Viewed 4357

When deploying container in kube-system you can see the fqdn of masternode api being used. When creating a namespace and deploy same container it is kubernetes service with internal ip for the KUBERNETES_SERVICE_HOST environment variable.

We cannot use kind: PodPreset in AKS so i don't know any other way on how to set this environment variable for new pods. Pods are being deploying with helm in this namespace so you can't set this environment variable in a way that helm will use this in deployment.

azure-aks kubernetes
2 Answers

The Kubernetes Container Environment provides several important resources to Containers, one of them is:

  • A list of all services that were running when a Container was created is available to that Container as environment variables. Those environment variables match the syntax of Docker links.

For a service named foo that maps to a Container named bar, the following variables are defined:

FOO_SERVICE_HOST=0.0.0.0 (The IP Address of the service foo)

FOO_SERVICE_PORT=65535 (the port of the service foo)

  • XXX_SERVICE_PORT is generated automatically based on the services available for the container at the moment of it's creation.

When deploying container in kube-system you can see the fqdn of masternode api being used. When creating a namespace and deploy same container it is kubernetes service with internal ip for the KUBERNETES_SERVICE_HOST environment variable.

  • It should work both ways because the kubernetes.default service is a relay agent to the master API, take a look at the description of the service:
$ k describe svc kubernetes
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP:                10.21.0.1
Port:              https  443/TCP
TargetPort:        443/TCP
Endpoints:         10.54.240.1:443
Session Affinity:  None
Events:            <none>

$ kubectl cluster-info
Kubernetes master is running at https://10.54.240.1
  • The endpoint of the kubernetes.default service is the master API IP, so if your deployment is not working as intended, it might have another issue under the hood.

You can also follow the instructions given in @djsly answer and open an issue on the prom-op github, vote for podPreset to become available on AKS or even experiment with other cloud providers (like GCP that offers a free tier so you can try out).

If you have any further questions let us know.

user_125241590

PodPreset is indeed not yet supported on AKS.

I would suggest that you vote on the following feedback: https://feedback.azure.com/forums/914020-azure-kubernetes-service-aks/suggestions/35054089-support-podpreset-alpha-feature

Right now, I would say the only way would be to augment the prometheus-operator chart with support for updating the ENV on the pods you need to overwrite. Pull Requests on Charts are often accepted quite fast.

https://github.com/helm/charts/tree/master/stable/prometheus-operator

or on any of the requirements defined in the chart that you think is missing the support.

user_26905250
Related