Disable Nginx ingress controller default backend

I'm using the Nginx ingress controller in a GKE cluster.

My nginx rule has a

spec:
  ingressClassName: nginx-example
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          ...

The proble is that if my application returns a 401 in one of the routes nginx is using the default backend and returning a 500 HTML page.

The app is supposed to be returning the code 401 and a json because the path being /

I want my app to be returning the response for all the codes and don't use the default backend.

I have tried to change Prefix to ImplementationSpecific. And adding nginx.ingress.kubernetes.io/rewrite-target: / as in here but nothing changes