Kubernetes Service get Connection Refused

Question

I am trying to create an application in Kubernetes (Minikube) and expose its service to other applications in same clusters, but i get connection refused if i try to access this service in Kubernetes node.

This application just listen on HTTP 127.0.0.1:9897 address and send response.

Below is my yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: exporter-test
  namespace: datenlord-monitoring
  labels:
    app: exporter-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: exporter-test
  template:
    metadata:
      labels:
        app: exporter-test
    spec:
      containers:
        - name: prometheus
          image: 34342/hello_world
          ports:
            - containerPort: 9897

---
apiVersion: v1
kind: Service
metadata:
  name: exporter-test-service
  namespace: datenlord-monitoring
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '9897'
spec:
  selector: 
    app: exporter-test
  type: NodePort  
  ports:
    - port: 8080
      targetPort: 9897
      nodePort: 30001

After I apply this yaml file, the pod and the service deployed correctly, and I am sure this pod works correctly, since when I login the pod by kubectl exec -it exporter-test-* -- sh, then just run curl 127.0.0.1:9897, I can get the correct response.

Also, if I run kubectl port-forward exporter-test-* -n datenlord-monitoring 8080:9897, I can get correct response from localhost:8080. So this application should work well.

However, when I trying to access this service from other application in same K8s cluster by exporter-test-service.datenlord-monitoring.svc:30001 or just run curl nodeIp:30001 in k8s node or run curl clusterIp:8080 in k8s node, I got Connection refused

Anyone had same issue before? Appreciate for any help! Thanks!

user_9250303 · Answer

you are mixing two things here. NodePort is the port the application is available from outside your cluster. Inside your cluster you need to access your service via the service port, not the NodePort.

Try changing exporter-test-service.datenlord-monitoring.svc:30001 to exporter-test-service.datenlord-monitoring.svc:8080

user_15537201 · Answer

Welcome to the community!

There are no issues with behaviour you observed. In short words kubernetes cluster (which is minikube in this case) has its own isolated network with internal DNS.

One way to access your service on the node: you specified nodePort for your service and this made the service accessible on the localhost:30001. You can check it by running on your host:

$ kubectl get svc -n datenlord-monitoring

NAME                    TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
exporter-test-service   NodePort   10.111.191.159           8080:30001/TCP   2m45s

# Test:

curl -I localhost:30001
HTTP/1.1 200 OK

Another way to expose service to the host network is to use minikube tunnel (run in the another console). You'll need to change service type from NodePort to LoadBalancer:

$ kubectl get svc -n datenlord-monitoring

NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)          AGE
exporter-test-service   LoadBalancer   10.111.191.159   10.111.191.159   8080:30001/TCP   18m

# Test:

$ curl -I 10.111.191.159:8080
HTTP/1.1 200 OK

Why some of options doesn't work.

Connection to the service by its DNS + NodePort. NodePort is used to link host IP and NodePort to service port inside kubernetes cluster. Internal DNS is not accessible outside kubernetes cluster (unless you don't add IPs to /etc/hosts on your host machine)

Inside the cluster you should use internal DNS with internal service port which is 8080 in your case. You can check how this works with a separate container in the same namespace (e.g. image curlimages/curl) and get following:

$ kubectl exec -it curl -n datenlord-monitoring -- curl -I exporter-test-service:8080
HTTP/1.1 200 OK

Or from the pod in a different namespace:

$ kubectl exec -it curl-default-ns -- curl -I exporter-test-service.datenlord-monitoring.svc:8080
HTTP/1.1 200 OK

I've attached useful links which help you to understand this difference.

Edit: DNS inside deployed pod

$ kubectl exec -it exporter-test-xxxxxxxx-yyyyy -n datenlord-monitoring -- bash

root@exporter-test-74cf9f94ff-fmcqp:/# cat /etc/resolv.conf 
nameserver 10.96.0.10
search datenlord-monitoring.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Useful links:

Kubernetes Service get Connection Refused

2 Answers