nginx tcp stream (k8s) - keep client connection open when upstream closes

user_14889983 0
Viewed 142

I have an application that accepts TCP traffic (not HTTP) and I'd like the ability to have the traffic load balanced to it. However, one requirement is that when a client makes a connection, we do not close that connection under any circumstances (ideally) since we are dealing with some clients with older technology.

I've set up the kubernetes nginx ingress controller, but it isn't behaving how I'm hoping/expecting. What I would like is: If the connection to one of the upstream servers closes, then the client connection remains open for some amount of time while nginx picks a new upstream server and starts sending data to it. I am not concerned about the stream's data being split across different upstream servers, I just need the connection to stay open from the client's perspective during something like a redeploy.

What is actually happening is that from my client's perspective, currently when the upstream server app closes the connection, my connection is closed and I have to reconnect.

The ingress controller has this configuration, which I thought would accomplish what I want, but it doesn't seem to be working as expected:


       server {
               preread_by_lua_block {
                       ngx.var.proxy_upstream_name="tcp-my-namespace-my-service-7550";
               }

               listen                  7550;

               proxy_timeout           600s;
               proxy_next_upstream     on;
               proxy_next_upstream_timeout 600s;
               proxy_next_upstream_tries   3;

               proxy_pass              upstream_balancer;

       }

Any help at all is greatly appreciated and I'm happy to provide more info.

kubernetes nginx networking
3 Answers

Unfortunately I haven't been able to get this functionality working with nginx. What I've ended up doing is writing my own basic TCP reverse-proxy that does what I need - if a connection to a backend instance is lost, it attempts to get a new one without interrupting the frontend connection. The traffic that we receive is fairly predictable in that I don't expect that moving the connection will interrupt any of the "logical" messages on the stream 99% of the time.

I'd still love to hear if anyone knows of an existing tool that has this functionality, but at the moment I'm convinced that there isn't one readily available.

user_148899830

What you describe is how nginx works out of the box with http. However

  1. Nginx has a detailed understanding of http
  2. HTTP is a message based protocol i.e. uses requests and replies

Since nginx knows nothing about the protocol you are using, even if it uses a request/reply mechanism with no implied state, nginx does not know whether it has received a request not to to replay it elsewhere.

You need to implement a protol-aware mitm.

user_2239920

I think you need to configure your Nginx Ingress to enable the keepalive options as listed in the documentation here. For instance in your nginx configuration as:

...
keepalive 32;
...

This will activate the keepalive functionality with a cache of upto 32 connections active at a time.

user_67133270
Related