Nginx ingress : Host based routing on TCP port

user_9233807 0
Viewed 702

Usage of the same TCP port for Rabbitmq 5672 and transfer requests to different namespaces/rabbitmq_service based on the host-based routing.

What works:

chart: nginx-git/ingress-nginx
version: 3.32.0
values:
  - tcp:
      5672: "cust1namespace/rabbitmq:5672"

Block reflected in nginx.conf:

server {
    preread_by_lua_block {
        ngx.var.proxy_upstream_name="tcp-cust1namespace-services-rabbitmq-5672";
    }
    listen                  :5672;
    proxy_pass              upstream_balancer;
}

Note: this will transfer all the requests coming to port 5672 to cust1namespace/rabbitmq:5672, irrespective of the client domain name and we want host-based routing based on domain name.

What is expected:

chart: nginx-git/ingress-nginx
version: 3.32.0
values:
  - tcp:
      cust1domainname:5672: "cust1namespace/rabbitmq:5672"
      cust2domainname:5672: "cust2namespace/rabbitmq:5672"

Error:

Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Service.spec.ports[3].port): invalid type for io.k8s.api.core.v1.ServicePort.port: got "string", expected "integer", ValidationError(Service.spec.ports[4].port): invalid type for io.k8s.api.core.v1.ServicePort.port: got "string", expected "integer"]

The final nginx.conf should look like:

server {
    preread_by_lua_block {
        ngx.var.proxy_upstream_name="tcp-cust1namespace-services-rabbitmq-5672";
    }
    listen                  cust1domainname:5672;
    proxy_pass              upstream_balancer;
}
    
server {
    preread_by_lua_block {
        ngx.var.proxy_upstream_name="tcp-cust2namespace-services-rabbitmq-5672";
    }
    listen                  cust2domainname:5672;
    proxy_pass              upstream_balancer;
}
kubernetes-helm kubernetes-ingress kubernetes nginx
1 Answers

A bit of theory

Approach you're trying to implement is not possible due to network protocols implementation and difference between them.

TCP protocol works on transport layer, it has source and destination IPs and ports, it does not have any hosts information within. In turn HTTP protocol works on application layer which seats on top of the TCP and it does have information about host where this request is intended to be sent.

Please get familiar with OSI model and protocols which works on these levels. This will help to avoid any confusion why this works this way and no other.

Also there's a good answer on quora about difference between HTTP and TCP protocols.

Answer

At this point you have two options:

  1. Use ingress to work on application layer and let it direct traffic to services based on hosts which are presented in request body. All traffic should go through ingress endpoint (usually it's loadbalancer which is exposed outside of the cluster).

Please find examples with

  1. Use ingress to work on transport layer and expose separate TCP ports for each service/customer. In this case traffic will be passed through ingress directly to services.

Based on your example it will look like:

chart: nginx-git/ingress-nginx
    version: 3.32.0
    values:
    - tcp:
        5672: "cust1namespace/rabbitmq:5672" # port 5672 for customer 1
        5673: "cust2namespace/rabbitmq:5672" # port 5673 for customer 2
        ...
user_155372010
Related