kubernetes-Ingress resource is not working as expected to expose 2 apps using subdomains

Answering questions in the post:

1: I was thinking that maybe I should have 2 nodes? one with mysql and app1 and the other one with mysql and app2? so in one I could access app1.localhost/cxf services and in the other app2.localhost/cxf services... maybe doesn't make much sense... and I was reading that I need kubeadm for that, and there is no way to install it on windows. I think I must use minikube for that instead?

You don't need to have 2 nodes. You can use single node Kubernetes installation for example minikube or Docker Desktop (with Kubernetes) to spawn your app1, app2 and your mysql. I will include the guide to expose example apps like you mentioned on the end of this answer.

As for kubeadm:

kubeadm performs the actions necessary to get a minimum viable cluster up and running. By design, it cares only about bootstrapping, not about provisioning machines.

Kubernetes.io: kubeadm

In simple words kubeadm is used on machines (bare-metal, VM's) to run the steps to provision the Kubernetes cluster (assuming that OS is installed, configured etc.)

A tip!

minikube is using kubeadm to provision it's single node cluster.

2: shouldn't I be able to create another loadBalancer? I wanted to, so the selector would be app2 in that loadBalancer, but since I already have one, the new one just stays until I remove the first one. But when I installed the nginx-ingress controller it came with a type loadBalancer, and if I create an yaml with the same content, changing the ports it will work and wont stay pending...why?

We will need to distinguish 2 terms here:

• service of type LoadBalancer - is used to expose your apps externally using a cloud provider's load balancer. In solution like minikube or docker it's usually mapped to the VM's IP (minikube) or localhost your machine (Docker)

• Ingress resource - exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

In my experience service of type LoadBalancer is mostly using Layer4 where Ingress resource is operating at Layer7. LoadBalancer won't route your traffic based on the Host header. It will route based on the labels and selectors of your service and your Deployment. Ingress resource is designed to serve HTTP/HTTPS traffic. It will incorporate tools that will route your traffic based on the Host header.

This are 2 separate ways to expose your apps and should be treated as different.

A tip!

nginx-ingress is using service of type LoadBalancer to route the traffic to nginx-ingress-controller. From there it's analyzed and sent to the specific destinations (according to Ingress resource)!

The fact that you have Pending in the External-IP column could be related to the fact that the ports used in the service of type LoadBalancer are already used (port parameter).

3: if I have 2 replicas of app1, and 2 replicas of app2, there should be a loadBalancer for each app right?

For each app you need to have a LoadBalancer. It doesn't matter how many replicas your app has (it can be even 100). Basing on previous sentence:

• app1 - 1st service of type LoadBalancer
• app2 - 2nd service of type LoadBalancer

I try to think like below when exposing apps (it's really simplified):

• I'm exposing traffic which is not HTTP/HTTPS - service of type LoadBalancer
• I'm exposing some example app/apps and I need to have functionalities of Ingress (Host based routing, rewrites, SSL, etc.) - Ingress resource

Disclaimer!

You don't need to use service of type LoadBalancer when exposing apps with Ingress resource.

4: Should I include a loadBalancer for mysql? does it make sense having mysql replicas?

There are a lot of useful guides on deploying mysql in Kubernetes. To give your app1 and app2 an access to your mysql pod you can use service of type ClusterIP or a headlessService. The multiple replicas of your mysql will definitely increase fault tolerance in multi node clusters but it would need to have some kind of replication logic already included.

You can read more by following below links:

• Kubernetes.io: Docs: Defining a service
• Kubernetes.io: Docs: Headless services
• Kubernetes.io: Docs: Run replicated stateful application

5: Since when I install nginx-controller, when I check my kubernetes dashboard there is a loadBalancer that came with it. Its ok to edit and add my karaf ports to expose the service?

Service of type LoadBalancer that was created when provisioning ingress-nginx-controller was specifically created only for ingress-nginx-controller.

If you would like to expose your karaf you have an option to:

• Expose it with ingress-nginx-controller according to the specification of your Ingress resource.
• Expose it with nginx-ingress-controller via TCP/UDP services (it will not have the features of Ingress resource):
• Expose it with another example of service of type LoadBalancer

This guide is designed to work with Docker Desktop with Kubernetes installed.

Assuming that you want to:

I should be able to access app1 version in app1.localhost:8181/cxf, and app2 version in app2.localhost:8181/cxf

Assuming that you have a working Kubernetes instance with your Docker Desktop you will need to follow below example:

• Install nginx-ingress
• Spawn hello-one example app and a service associated with it
• Spawn hello-two example app and a service associated with it
• Edit the nginx-ingress service of type LoadBalancer
• Create and apply an Ingress resource
• Make some DNS changes
• Test

Install nginx-ingress

Run:

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.35.0/deploy/static/provider/cloud/deploy.yaml

Disclaimer!

1. There could be differences when trying to follow this steps between Windows and Mac based environments
2. There could be differences when trying to follow this steps on minikube instances

Installation page:

• Kubernetes.github.io: Ingress-nginx: Deploy

Spawn hello-one example app and a service associated with it

Here is the definition for hello-one app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-one
spec:
  selector:
    matchLabels:
      app: hello-one
      version: 1.0.0
  replicas: 1
  template:
    metadata:
      labels:
        app: hello-one
        version: 1.0.0
    spec:
      containers:
      - name: hello
        image: "gcr.io/google-samples/hello-app:1.0" # <-- notice the 1.0
        env:
        - name: "PORT"
          value: "50001"

Here is the definition for a service for hello-one app:

apiVersion: v1
kind: Service
metadata:
  name: hello-one-service
spec:
  type: NodePort
  selector:
    app: hello-one
    version: 1.0.0
  ports:
  - name: hello-one-port
    protocol: TCP
    port: 80
    targetPort: 50001

Spawn hello-two example app and a service associated with it

Here is the definition for hello-two app:

apps/v1
kind: Deployment
metadata:
  name: hello-two
spec:
  selector:
    matchLabels:
      app: hello-two
      version: 2.0.0
  replicas: 1
  template:
    metadata:
      labels:
        app: hello-two
        version: 2.0.0
    spec:
      containers:
      - name: hello
        image: "gcr.io/google-samples/hello-app:2.0" # <-- notice the 2.0
        env:
        - name: "PORT"
          value: "50001"

Here is the definition for a service for hello-two app:

apiVersion: v1
kind: Service
metadata:
  name: hello-two-service
spec:
  type: NodePort
  selector:
    app: hello-two
    version: 2.0.0
  ports:
  - name: hello-two-port
    protocol: TCP
    port: 80
    targetPort: 50001

Edit the nginx-ingress service of type LoadBalancer

You will need to edit the service of type LoadBalancer of your nginx-ingress-controller to support ports other than 80 and 443.

• $ kubectl edit service -n ingress-nginx ingress-nginx-controller

The part responsible for routing:

  ports:
  - name: http
    nodePort: 32202
    port: 80 # <-- CHANGE HERE FOR 8181!
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 31130
    port: 443 # <-- CHANGE HERE FOR 8182!
    protocol: TCP
    targetPort: https

Create and apply an Ingress resource

You will need to use the following Ingress resource to have an access to your apps on:

• hello-one.kubernetes.local:8181/cxf
• hello-two.kubernetes.lcoal:8181/cxf

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-ingress 
spec:
  rules:
  - host: hello-one.kubernetes.docker.internal
    http:
      paths:
      - path: /
        backend:
          serviceName: hello-one-service 
          servicePort: hello-one-port
  - host: hello-two.kubernetes.docker.internal
    http:
      paths:
      - path: /
        backend:
          serviceName: hello-two-service 
          servicePort: hello-two-port

Disclaimer!

This Ingress doesn't have the /cxf path configured. You can look here for more resources:

• Kubernetes.github.io: Ingress path matching

Make some DNS changes

You will need to edit the DNS setting of your host to support newly created host names. This part will depend on the OS you use. Lines to add:

127.0.0.1 hello-one.kubernetes.docker.internal
127.0.0.1 hello-two.kubernetes.docker.internal

Test

You can test your setup by either curl or your web browser:

• $ curl hello-one.kubernetes.docker.internal:8181

Hello, world!
Version: 1.0.0
Hostname: hello-one-696db54c4d-5fqt8

• $ curl hello-two.kubernetes.docker.internal:8181

Hello, world!
Version: 2.0.0
Hostname: hello-two-664ddf4cf6-4jhfr