Ingress nginx routing in kubernetes (convertign from docker-compose with nginx reverse proxy)

user_12246875 0
Viewed 689

I'm trying to move from running some services in docker-compose to kubernetes, and struggling with the move from nginx reverse proxy to ingress nginx. The service uses h2o ai which has a web interface. I'm failing to bring up the web interface in the k8s version.

I think that I need to amend the spec in the ingress file to route appropriately, grateful for any pointers on how to do this.

In the docker solution with nginx as reverse proxy, when I access http://k8s-master:3002 it redirects to http://k8s-master:3002/flow/index.html and displays correctly

For k8s,(I have the ingress-nginx-controller running as NodePort on 32000) , I try to access https://k8s-master:32000/h2otest and get an error message as follows:

{"__meta":{"schema_version":3,"schema_name":"H2OErrorV3","schema_type":"H2OError"},"timestamp":1602674406013,"error_url":"Resource /h2otest","msg":"\n\nERROR MESSAGE:\n\nResource /h2otest not found\n\n","dev_msg":"\n\nERROR MESSAGE:\n\nResource /h2otest not found\n\n","http_status":404,"values":{},"exception_type":"water.exceptions.H2ONotFoundArgumentException","exception_msg":"\n\nERROR MESSAGE:\n\nResource /h2otest not found\n\n","stacktrace":["water.exceptions.H2ONotFoundArgumentException: Resource /h2otest not found","    water.api.RequestServer.response404(RequestServer.java:743)","    water.api.RequestServer.getResource(RequestServer.java:846)","    water.api.RequestServer.serve(RequestServer.java:465)","    water.api.RequestServer.doGeneric(RequestServer.java:301)","    water.api.RequestServer.doGet(RequestServer.java:225)","    javax.servlet.http.HttpServlet.service(HttpServlet.java:687)","    javax.servlet.http.HttpServlet.service(HttpServlet.java:790)","    org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)","    org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)","    org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)","    org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)","    org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)","    org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)","    org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)","    org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)","    org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)","    org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)","    org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)","    water.webserver.jetty9.Jetty9ServerAdapter$LoginHandler.handle(Jetty9ServerAdapter.java:130)","    org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)","    org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)","    org.eclipse.jetty.server.Server.handle(Server.java:531)","    org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)","    org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)","    org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)","    org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)","    org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)","    org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)","    org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)","    java.lang.Thread.run(Thread.java:748)"]}

UPDATE: based on Ken Chen's comment

I've changed the ingress below to use nginx.ingress.kubernetes.io/rewrite-target: /$2

Now when I type https://k8s-master:32000/h2otest it redirects to: https://k8s-master:32000/flow/index.html

Typing https://k8s-master:32000/h2otest/flow/index.html directly works. How do I amend the rule so that https://k8s-master:32000/h2otest goes directly to https://k8s-master:32000/h2otest/flow/index.html ?

The following contains the files for each version:

docker version

docker-compose.yml

services :
  test-h2o-svc:
    build:
      context: h2o
      dockerfile: Dockerfile
    image: test-h2o:3.30.1.1
    restart: always
    networks:
      - testnet
#    ports:
#      - 54323:54321

  test-reverse-proxy:
    image: nginx:alpine
    hostname: reverse-proxy
    restart: always
    ports:
      - 3002:3002
    volumes:
      - "./nginx_files/nginx.conf:/etc/nginx/nginx.conf:rw"
      - "./nginx_files/sites-available/test.conf:/etc/nginx/conf.d/sites-available/test.conf:rw"
      - "./nginx_files/sites-available/test.conf:/etc/nginx/conf.d/sites-enabled/test.conf:rw"
    networks:
      - testnet

networks:
 testnet:
version: "3.5"

nginx.conf

user  nginx;
worker_processes  4;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}



http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;
    keepalive_timeout  180;
    #gzip  on;

    include /etc/nginx/conf.d/sites-enabled/*.conf;
    client_body_buffer_size     50M;
    client_max_body_size 2000M;
}

test.conf

upstream link_h2o-svc {
  server        test-h2o-svc:54321;
}

server {
  listen        3002;
  location / {
    proxy_pass  http://link_h2o-svc;
   }
}

Dockerfile for container: test-h2o:3.30.1.1

FROM ubuntu:latest

ARG H2O_VERSION=3.30.1.1

RUN apt-get update \
    && apt-get install default-jdk unzip wget -y

RUN wget https://h2o-release.s3.amazonaws.com/h2o/rel-zeno/1/h2o-${H2O_VERSION}.zip \
    && unzip h2o-${H2O_VERSION}.zip

ENV H2O_VERSION ${H2O_VERSION}
CMD java -jar h2o-${H2O_VERSION}/h2o.jar
k8s version 
kind: Namespace
apiVersion: v1
metadata:
  name: ns-test
  labels:
    name: ns-test
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: /snap/kompose/19/kompose-linux-amd64 convert
    kompose.version: 1.21.0 (992df58d8)
  creationTimestamp: null
  labels:
    app: test-h2o
  name: test-h2o
  namespace: ns-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-h2o
  
  strategy: {}
  template:
    metadata:
      annotations:
        kompose.cmd: /snap/kompose/19/kompose-linux-amd64 convert
        kompose.version: 1.21.0 (992df58d8)
      creationTimestamp: null
      labels:
        app: test-h2o
    spec:
      containers:
      - image: test-h2o:3.30.1.1
        imagePullPolicy: ""
        name: test-h2o
        resources: {}
      restartPolicy: Always
      serviceAccountName: ""
      volumes: null
status: {}
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: /snap/kompose/19/kompose-linux-amd64 convert
    kompose.version: 1.21.0 (992df58d8)
  creationTimestamp: null

  name: test-h2o-svc
  namespace: ns-test
spec:
  ports:
  - name: "54321"
    port: 54321
    targetPort: 54321
  selector:
    app: test-h2o
status:
  loadBalancer: {}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: ns-test
  annotations:
    #ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - http:
      paths:
        - path: /h2otest(/|$)(.*)
          pathType: Prefix
          backend:
            service:
              name: test-h2o-svc
              port: 
                number: 54321
h2o nginx-ingress kubernetes
2 Answers

How do I amend the rule so that https://k8s-master:30439/h2otest goes directly to https://k8s-master:30439/h2otest/flow/index.html

That is not an nginx-ingress question. Like most of the other ingresses nginx-ingress is there to do name based virtual hosting and SNI (or SSL Name based virtual hosting) with SSL offloading. Out of the box it's limited in any more.

While nginx-ingress can do some basic re-writing, it's limited to rewriting /path/(.*) to /(.*) and passing it to the next container to handle the traffic from there. If you're wanting it to do more you're into writing config maps for nginx-ingress. Yes you'll want to set annotations for client body size etc, but the app itself (not the ingress) needs to be handling the /h2otest -> /h2otest/flow/index.html rewriting (or / -> /flow/index.html).

From what you're describing your layout looks like (nginx)ingress -> app

You may need to change the layout to (nginx)ingress -> nginx ->app with the middle nginx doing the / -> /flow/index.html rewrite

A word of advice. If you have any chance use name based virtual hosting rather than path virtual hosting (unless you have a good reason eg cors) for path based. Alot of apps these days just assume they are on / an the config gets "interesting" if you're elsewhere

user_56049350

Nginx reverse proxy rewrite /h2otest to / but h2o redirect / to /flow/index.html. At this point, nginx does not have a rule to handle /flow so result to 404.

Try remove the path and use root / instead.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: ns-test
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            serviceName: test-h2o-svc
            servicePort: 54321
user_50391170
Related