Kubernetes: Not able to communicate within two services (different pod, same namespace)

user_13956954 0
Viewed 841

I am not able to communicate between two services.

post-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-data-deployment
  labels:
spec:
  replicas: 1
  selector:
   matchLabels:
    app: python-web-selector
    tier: backend
  template:
   metadata:
     labels:
       app: python-web-selector
       tier: backend
   spec:
    containers:
    - name: python-web-pod
      image: sakshiarora2012/python-backend:v10
      ports:
      - containerPort: 5000

post-deployment2.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-data-deployment2
  labels:
spec:
  replicas: 1
  selector:
   matchLabels:
    app: python-web-selector2
    tier: backend
  template:
   metadata:
     labels:
       app: python-web-selector2
       tier: backend
   spec:
    containers:
    - name: python-web-pod2
      image: sakshiarora2012/python-backend:v8
      ports:
      - containerPort: 5000

post-service.yml

apiVersion: v1
kind: Service
metadata:
  name: python-data-service
spec:
  selector:
   app: python-web-selector
   tier: backend
  ports:
      - port: 5000
        nodePort: 30400
  type: NodePort

post-service2.yml

apiVersion: v1
kind: Service
metadata:
  name: python-data-service2
spec:
  selector:
   app: python-web-selector2
   tier: backend
  ports:
      - port: 5000
  type: ClusterIP

When I go and try to ping from 1 container to another, it is not able to ping

root@python-data-deployment-7bd65dc685-htxmj:/project# ping python-data-service.default.svc.cluster.local
PING python-data-service.default.svc.cluster.local (10.107.11.236) 56(84) bytes of data.
^C
--- python-data-service.default.svc.cluster.local ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 139ms

If I see dns entry it is showing

sakshiarora@Sakshis-MacBook-Pro Student_Registration % kubectl exec -i -t dnsutils -- nslookup  python-data-service 
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   python-data-service.default.svc.cluster.local
Address: 10.107.11.236

sakshiarora@Sakshis-MacBook-Pro Student_Registration % 
sakshiarora@Sakshis-MacBook-Pro Student_Registration % kubectl exec -i -t dnsutils -- nslookup  python-data-service2
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   python-data-service2.default.svc.cluster.local
Address: 10.103.97.40


sakshiarora@Sakshis-MacBook-Pro Student_Registration % kubectl get pod -o wide 
NAME                                       READY   STATUS    RESTARTS   AGE     IP           NODE       NOMINATED NODE   READINESS GATES
dnsutils                                   1/1     Running   0          5m54s   172.17.0.9   minikube   <none>           <none>
python-data-deployment-7bd65dc685-htxmj    1/1     Running   0          47m     172.17.0.6   minikube   <none>           <none>
python-data-deployment2-764744b97d-mc9gm   1/1     Running   0          43m     172.17.0.8   minikube   <none>           <none>
python-db-deployment-d54f6b657-rfs2b       1/1     Running   0          44h     172.17.0.7   minikube   <none>           <none>

sakshiarora@Sakshis-MacBook-Pro Student_Registration % kubectl describe svc python-data-service
Name:                     python-data-service
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"python-data-service","namespace":"default"},"spec":{"ports":[{"no...
Selector:                 app=python-web-selector,tier=backend
Type:                     NodePort
IP:                       10.107.11.236
Port:                     <unset>  5000/TCP
TargetPort:               5000/TCP
NodePort:                 <unset>  30400/TCP
Endpoints:                172.17.0.6:5000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
sakshiarora@Sakshis-MacBook-Pro Student_Registration % kubectl describe svc python-data-service2
Name:              python-data-service2
Namespace:         default
Labels:            <none>
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"python-data-service2","namespace":"default"},"spec":{"ports":[{"p...
Selector:          app=python-web-selector2,tier=backend
Type:              ClusterIP
IP:                10.103.97.40
Port:              <unset>  5000/TCP
TargetPort:        5000/TCP
Endpoints:         172.17.0.8:5000
Session Affinity:  None
Events:            <none>

sakshiarora@Sakshis-MacBook-Pro Student_Registration %

I think if in DNS table it show if of range 172,17.0.X then it will work, but not sure why it is not showing in dns entry, Any pointers?

kube-dns minikube kubernetes
5 Answers

Ping doesn't work on services ClusterIP addresses because they are from virtual addresses created by iptables rules that redirect packets to the endpoints(pods).

You should be able to ping a pod, but not a service.

You can use curl or wget

For example wget -qO- POD_IP:80

or You can try

wget -qO- http://your-service-name:port/yourpath
curl POD_IP:port_number
user_80537140

If you want to access python-data-service from outside the cluster using NodePort and you are using minikube you should be able to do so by using curl $(minikube service python-data-service --url) from anywhere outside the cluster i.e from your system

If you want to communicate between two microservice within the cluster then simply use ClusterIP type service instead of NodePort type.

To identity if it's a service issue or pod issue use PODIP directly in the curl command. From the output of kubectl describe svc python-data-service the Pod IP for service python-data-service is 172.17.0.6. So try curl 172.17.0.6:5000/getdata

user_18394820

In order to start debugging your services I would suggest the following steps:

Check that your service 1 is accessible as a Pod:

kubectl run test1 -it --rm=true --image=busybox --restart=Never -n default -- wget -O - http://172.17.0.6:5000

Check that your service 2 is accessible as a Pod:

kubectl run test2 -it --rm=true --image=busybox --restart=Never -n default -- wget -O - 172.17.0.8:5000

Then, check that your service 1 is accessible as a Service using the corresponding cluster IP and then DNS Name:

kubectl run test2 -it --rm=true --image=busybox --restart=Never -n default -- wget -O - 10.107.11.236:5000

kubectl run test2 -it --rm=true --image=busybox --restart=Never -n default -- wget -O - http://python-data-service:5000

Then, check that your service 2 is accessible as a Service using the corresponding cluster IP and then DNS Name:

kubectl run test2 -it --rm=true --image=busybox --restart=Never -n default -- wget -O - 10.103.97.40:5000

kubectl run test2 -it --rm=true --image=busybox --restart=Never -n default -- wget -O - http://python-data-service2:5000

Then, if needed, check that your service 2 is accessible through your node port (you would need to know the IP Address of the Node where the service has been exposed, for instance in minikube it should work:)

wget -O - http://192.168.99.101:30400

From your Service manifest I can recommend as a good practice to specify both port and targetPort as you can see at

https://canterafonseca.eu/kubernetes/certification/application/developer/cncf/k8s/cloud/native/computing/ckad/deployments/services/preparation-k8s-ckad-exam-part4-services.html#-services

On the other hand if you only need to expose to the outside world one of the services you can create a headless service (see also my blog post above).

user_14337530

Are you able to connect to your pods maybe try port-forward to see if you can connect & then check the connectivity in two pods. Last check if there is default deny network policy set there - maybe you have some restrictions at network level.

kubectl get networkpolicy -n <namespace>
user_61433320

Try to look into the logs using kubectl logs PODNAME so that you know what's happening. From first sight, I think you need to expose the ports of both services: kubectl port-forward yourService PORT:PORT.

user_66517630
Related