Backend Services quota on GKE

Question

My question is regarding quotas of Google Kubernetes Engine.

I have an instance running 4 pods, each pod is referring to a microservice (api) containing 3 containers:

Spring Boot App
esp: endpoints
cloudsqlproxy

For each pod (microservice), I have a deployment yaml which includes a nodeport service. Along with that, there is an ingress mapping all these services. Now I need to deploy another microservice (pod with same 3 containers), but the quota of 5 backend services is in the limit.

I dont know if I'm doing something wrong or this quota is very small. I think, four microservices is very little for a technology that supports this approach.

So, Am I missing something in this architecture / configuration? Something that I'm doing wrong?

Here is my Ingress configuration:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: sigma-ingress
  annotations:
    # type of controller to use:
    kubernetes.io/ingress.class: "gce"
    # CORS
    ingress.kubernetes.io/enable-cors: "true"
    #
    # ingress.kubernetes.io/rewrite-target: /
    # Don't rediret to HTTPS
    ingress.kubernetes.io/ssl-redirect: "false"
    # Block HTTP requests
    kubernetes.io/ingress.allow-http: "false"
spec:
  tls:
  - secretName: sigma-ssl
  rules:
  - http:
      paths:
      - path: /agro/*
        backend:
          serviceName: api-agro
          servicePort: 443
      - path: /fazendas
        backend:
          serviceName: api-fazenda
          servicePort: 443
      - path: /fazendas/*
        backend:
          serviceName: api-fazenda
          servicePort: 443
      - path: /clima
        backend:
          serviceName: api-clima
          servicePort: 443
      - path: /clima/*
        backend:
          serviceName: api-clima
          servicePort: 443
      - path: /ocorrencias
        backend:
          serviceName: api-inspecao
          servicePort: 443
      - path: /ocorrencias/*
        backend:
          serviceName: api-inspecao
          servicePort: 443

Thanks in advance

Peter

user_2989261 · Answer

You are probably hitting the GCP Load balancer backend limit which is usually low (mine is 9) This if you several other load balancers with configured backends that are making you hit that limit and you are mentioning that you already have other NodePort services (are they using another load balancer with a different backend?)

Your ingress should use just one backend that points to all the nodes in your cluster.

Besides the gcloud command line, you can see the limit from the UI.

You can also see the details on the load balancer that is serving your Ingress.

And you can see all your load balancer too:

You can request a backend quota increase from GCP, or you can remove unused load balancers.

user_9705485 · Answer

You can check the list of backends with gcloud compute backend-services list or go to the quotas page in the gcloud web console and it should show you both your allowed usage and current usage. You should also be able to see there at what level it is set from (it may come from a parent project policy). You can also request an increase from the same page using the edit button.

Backend Services quota on GKE

2 Answers