How to set the private ip address of AWS NLB when create Kubenetes service?

user_6687534 0
Viewed 1900

I'd like to create a nginx ingress controller with AWS internal NLB, the requirement is fix the IP address of NLB endpoint, for example, currently the NLB dns of Nginx ingress service is abc.elb.eu-central-1.amazonaws.com which is resolved to ip address 192.168.1.10, if I delete and re-create nginx ingress controller, I want the NLB DNS must be the same as before. Having a look in kubernetes service annotation, I did not see any way to re-use existing NLB, however, I find out the annotation service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses in link, as far as I understand that it allow me to set ip address for NLB, but it not work as my expectation, everytime I re-created nginx controller, the ip address is difference, Below is K8s service yaml file.

# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: "10.136.103.251"
    service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-00df069133b22"
  labels:
    helm.sh/chart: ingress-nginx-3.23.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local

I know this requirement is werid, is it possible to do that?

aws-nlb nginx-ingress kubernetes amazon-web-services
2 Answers

If your Kubernetes cluster runs on a VPC with more than one subnet (which is probably the case), you must provide a private ip address for each subnet.

I installed the AWS Load balancer controller with the helm chart, then i installed the nginx ingress controller with this helm chart :

helm install nginx-ingress ingress-nginx/ingress-nginx --namespace nginx-ingress -f internal-ingress-values.yaml

Here the content of internal-ingress-values.yaml

controller:
  ingressClass: nginx
  service:
    enableHttp: false
    enableHttps: true
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: external
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      service.beta.kubernetes.io/aws-load-balancer-scheme: internal
      service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: 10.136.103.251, 10.136.104.251
      service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-00a1a7f9949aa0ba1, subnet-12ea9f1df24aa332c
  ingressClassResource:
    enabled: true
    default: true

According to the documentation the service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses annotation length/order must match subnets

So, you must provide IP addresses and subnet in the same order (don't mismatch). If you take my example above, you must make sure that :

  • 10.136.103.251 is included in subnet-00a1a7f9949aa0ba1
  • 10.136.104.251 is included in subnet-12ea9f1df24aa332c

It's a good idea to tag your subnets according to the documentation :

Key: kubernetes.io/cluster/my-cluster-name Value: shared

Key: kubernetes.io/role/internal-elb Value: 1

I tested this K8S on 1.20 and it works for my project. Don't provide "ingressClassResource" if you're on K8S <= 1.17.

user_165418670

The only LBs that will be managed (at least at the current version 2.3 of the AWS LB Controller) are "nlb-ip" and "external" types. This is specified at: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/guide/service/annotations/#legacy-cloud-provider

The annotation service.beta.kubernetes.io/aws-load-balancer-type is used to determine which controller reconciles the service. If the annotation value is nlb-ip or external, legacy cloud provider ignores the service resource (provided it has the correct patch) so that the AWS Load Balancer controller can take over. For all other values of the annotation, the legacy cloud provider will handle the service. Note that this annotation should be specified during service creation and not edited later.

user_76530220
Related