503 response code when called from inside cluster

user_11567589 0
Viewed 367

I have deployed a demo service (running on port 8000) in our K8s environment which has Istio installed (1.5.6, default profile). When I make a call from outside the cluster to the public address, it succeeds. When I make a call from a pod inside the cluster to the internal cluster address, it fails with response code 503.

When I change my Virtual Service to use the port instead of the subset, then it succeeds in both cases (external and internal call).

Any ideas what I'm doing wrong?

apiVersion: v1
kind: Namespace
metadata:
  labels:
    dgp-origin: demo-app
    istio-injection: enabled
  name: demo
---
apiVersion: v1
kind: Service
metadata:
  name: demo
  namespace: demo
  labels:
    app: demo
    version: v1
  annotations:
    networking.istio.io/exportTo: "*"
spec:
  ports:
  - name: http
    port: 8000
  selector:
    app: demo
    version: v1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  namespace: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app: demo
        version: v1
    spec:
      containers:
      - name: echo
        image: paddycarey/go-echo
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8000
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: demo
  namespace: demo
spec:
  exportTo:
  - "*"
  host: demo.demo.svc.cluster.local
  subsets:
  - name: v1
    labels:
      app: demo
      version: v1
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: demo
  namespace: demo
spec:
  selector:
    app: istio-ingressgateway
  servers:
  - hosts:
    - demo.external.com
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
  - hosts:
    - demo.demo.svc.cluster.local
    port:
      name: http
      number: 80
      protocol: HTTP
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: demo
  namespace: demo
spec:
  exportTo:
  - "*"
  hosts:
  - demo.external.com
  - demo.demo.svc.cluster.local
  gateways:
  - mesh
  - demo/demo
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: demo.demo.svc.cluster.local
#        port: 
#          number: 8000
        subset: v1
    timeout: 55s

log info (from istio-proxy of another container)

external call: OK

{
    "authority": "-",
    "bytes_received": "511",
    "bytes_sent": "4744",
    "downstream_local_address": "172.19.2.100:443",
    "downstream_remote_address": "172.18.140.129:37992",
    "duration": "43",
    "istio_policy_status": "-",
    "method": "-",
    "path": "-",
    "protocol": "-",
    "request_id": "-",
    "requested_server_name": "-",
    "response_code": "0",
    "response_flags": "-",
    "route_name": "-",
    "start_time": "2020-08-10T10:32:25.149Z",
    "upstream_cluster": "PassthroughCluster",
    "upstream_host": "172.19.2.100:443",
    "upstream_local_address": "172.18.140.129:37994",
    "upstream_service_time": "-",
    "upstream_transport_failure_reason": "-",
    "user_agent": "-",
    "x_forwarded_for": "-"
}

internal call : NOT OK

{
    "authority": "demo.demo.svc.cluster.local",
    "bytes_received": "0",
    "bytes_sent": "0",
    "downstream_local_address": "172.18.212.107:80",
    "downstream_remote_address": "172.18.140.129:37802",
    "duration": "0",
    "istio_policy_status": "-",
    "method": "GET",
    "path": "/",
    "protocol": "HTTP/1.1",
    "request_id": "f875b032-f7d4-4f36-9ce1-38166aced074",
    "requested_server_name": "-",
    "response_code": "503",
    "response_flags": "NR",
    "route_name": "-",
    "start_time": "2020-08-10T10:33:51.262Z",
    "upstream_cluster": "-",
    "upstream_host": "-",
    "upstream_local_address": "-",
    "upstream_service_time": "-",
    "upstream_transport_failure_reason": "-",
    "user_agent": "curl/7.61.1",
    "x_forwarded_for": "-"
}

UPDATE : When service is on port 80 it works

apiVersion: v1
kind: Service
metadata:
  name: demo
  namespace: demo
  labels:
    app: demo
    version: v1
  annotations:
    networking.istio.io/exportTo: "*"
spec:
  ports:
  - name: http
    port: 80
    targetPort: 8000
  selector:
    app: demo
    version: v1
istio kubernetes
1 Answers

Based on the istio bookinfo app I would say the issue here are the missing labels in your deployment.

There is an productpage example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: details-v1
  labels:
    app: details
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: details
      version: v1
  template:
    metadata:
      labels:
        app: details
        version: v1
    spec:
      serviceAccountName: bookinfo-details
      containers:
      - name: details
        image: docker.io/istio/examples-bookinfo-details-v1:1.16.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080

Could you try to use your deployment after my edit?

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  namespace: demo
  labels:
    app: demo
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo
      version: v1
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app: demo
        version: v1
    spec:
      containers:
      - name: echo
        image: paddycarey/go-echo
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8000

EDIT

I have test your yamls, and additionally I have created my own example with nginx pod.

I have the same issue as you, mesh internall call works only if I add port 8000 to virtual service.

In my example with nginx everything works just fine.

So based on that I assume there is either something wrong with

  • paddycarey/go-echo image, as far as I checked, last time it was updated 4 years ago.
  • mesh gateway require port in virtual service, if it's other port than 80.

There are my yamls to test with nginx.

apiVersion: v1
kind: Namespace
metadata:
  labels:
    istio-injection: enabled
  name: demo-app

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-v1
  namespace: demo-app
spec:
  selector:
    matchLabels:
      app: nginx1
      version: v1
  replicas: 1
  template:
    metadata:
      labels:
        version: v1
        app: nginx1
    spec:
      containers:
      - name: nginx1
        image: nginx
        ports:
        - containerPort: 80
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh", "-c", "echo Hello nginx1 > /usr/share/nginx/html/index.html"]

---

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: demo-app
  labels:
    app: nginx1
spec:
  ports:
  - name: http-front
    port: 80
    protocol: TCP
  selector:
    app: nginx1

---

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: simpleexample
  namespace: demo-app
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP

---

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginxvirt
  namespace: demo-app
spec:
  gateways:
  - simpleexample
  - mesh
  hosts:
  - 'nginx.demo-app.svc.cluster.local'
  - 'example.com' 
  http:
  - route:
    - destination:
        host: nginx
        subset: v1

  
---

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: nginxdest
  namespace: demo-app
spec:
  host: nginx
  subsets:
  - name: v1
    labels:
      version: v1

---

apiVersion: v1
kind: Pod
metadata:
  name: ubu1
  namespace: demo-app
spec:
  containers:
  - name: ubu1
    image: ubuntu
    command: ["/bin/sh"]
    args: ["-c", "apt-get update && apt-get install curl -y && sleep 3000"]

External call test

curl -v -H "host: example.com" xx.xx.xx.xx/
HTTP/1.1 200 OK
Hello nginx1

Internal call test

root@ubu1:/# curl nginx/
Hello nginx1

Let me know if that was it or do you need further help.

user_119777600
Related