Kubernetes NGINX Ingress with Angular return wrong MIME-Type

Question

I have an Angular app deployed to kubernetes with the kubernetes ingress controller. There is one setup with a single-node cluster and one with a multi-node cluster. The MIME-Type problem occurs with both setups but the single-node can resolve it very quick while on the multi-node the latency is not acceptable.

On the multi-node cluster the Time To First Byte takes about 5 sec for:

initial call to example.com
runtime-es2015.js
polyfills-es2015.js
appConfig.json (custom config file)
favicon.ico
various png/svg files

What works within a normal timeframe are:

main-es2015.js
scripts.js
styles.css
ng-validate.js

My cluster setup is as following:

2 Control-Plane nodes
2 Worker nodes
Cluster networking with canal
The cluster was setup with RKE (if that matters)

The index.html in the Angular app contains:

Ingress for multi-node:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: multi-node-ingress
  namespace: non-default-namespace
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: example.com
    http:
      paths:
        - path: /?(.*)
          backend:
            serviceName: ng-app-name
            servicePort: 80

At my DNS provider I added A entries to both of my worker-nodes to directly forward all traffic to them. No Load Balancing is done on Layer 4.

With the help of a tutorial from Open Shift I was able to get a .pcap capturing which I could analyze in wireshark. The Content-Type is text/html for a call to e.g. http://example.com/favicon.svg and this results in a 404 Not Found. However after about 5 sec (never lower than that threshhold) it sometimes gets resolved.

My nginx-config for the container looks like this:

user  nginx;
worker_processes  auto;
...

events {
  worker_connections  1024;
}

http {
  include       /etc/nginx/mime.types;

  server {
    listen 80;

    location / {
      root /usr/share/nginx/html;
      try_files $uri $uri/ /index.html;
    }

    gzip on;
  }
}

What I noticed is that if I remove the line include /etc/nginx/mime.types; the page load fails entirely with Wrong MIME-Type error. But if it's present it takes around 5.02 to 5.15 secs (never under 5 secs) till it finds the resource.

There is an issue on github which might be related to my problem but the problem there is that nothing is loaded at all (wrong MIME-Type). I do get the page loaded, it's just painfully slow.

One suggestion of above Github issue is to add the custom type module which I did with a Config Map:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
data:
  http-snippet: |
    types {
      text/javascript .js;
      module .js;
    }
  default-type: application/octet-stream

The types are added to the nginx.conf of the ingress controllers but have no effect.
The default-type is not picked up and is still text/html.

Versions:

Kubernetes 1.18
Angular 10
nginx:1.19.3-alpine for hosting Angular app inside the container

user_10971917 · Answer

DNS responses are typically cached so in case you do two requests in row and second still has same Time to first Byte you can scratch out DNS latency. You can verify DNS latency by Wireshark.

Different namespace hardly makes any difference.

You didn't say any details about cloud provider but I assume there is load balancer (possibly offloading SSL decryption) balancing requests between 2 worker node (where ingress listens). You can enable some sort of logging for sure so do that.

Your nginx ingress is keeping access logs too so you can check them kubectl logs -n nginx-ingress - you'll have to run this on each of your worker node (or each node where ingress pods are located). Inside of these logs you'll find how long it took to nginx to receive response from your Angular App pod.

With combination of Wireshark, load balancer logs, nginx ingress logs and Angular logs (maybe you need to increase logging level for these to see every HTTP request) you should be able to pinpoint where the problem is.

user_1453887 · Answer

The issue is that the files are not loaded from the same Node and the retry mechanism takes around 5 sec. This can be fixed by adding node-affinity annotations like this:

nginx.ingress.kubernetes.io/affinity-mode: "persistent"
nginx.ingress.kubernetes.io/affinity: "cookie"

With this annotations every subsequent request will hit the same node. This can be a problem for high traffic sites but for us this works just fine.

For more details see the nginx docs about node-affinity.

Kubernetes NGINX Ingress with Angular return wrong MIME-Type

2 Answers