I have microservices running within EKS 1.22. Is there a possible way to check the microservice communication also pod-to-pod communication within the cluster through CLI?
1 Answers
Everything out of the box should work properly (assuming you have AWS EKS).
I think this article - Debug Services has very helpful hints.
Let's first check pod to pod communication using trival method - ping command. I created two NGINX deployments (one in default namespace, second one in namespace test):
kubectl create deployment nginx --image=nginx
kubectl create deployment nginx --image=nginx -n test
Now I will check IP addresses of both of them:
user@shell:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP
nginx-6799fc88d8-jxpj4 1/1 Running 0 3m13s 172.17.0.2
And also in test namespace:
user@shell:~$ kubectl get pods -o wide -n test
NAME READY STATUS RESTARTS AGE IP
nginx-6799fc88d8-z2glk 1/1 Running 0 103s 172.17.0.4
Now I will execute into one pod and check connectivity to the second one:
user@shell:~$ kubectl exec -it nginx-6799fc88d8-jxpj4 -- sh
# apt update
Hit:1 http://security.debian.org/debian-security bullseye-security InRelease
...
All packages are up to date.
# apt-get install inetutils-ping
Reading package lists... Done
...
Setting up inetutils-ping (2:2.0-1) ...
# ping 172.17.0.4
PING 172.17.0.4 (172.17.0.4): 56 data bytes
64 bytes from 172.17.0.4: icmp_seq=0 ttl=64 time=0.058 ms
64 bytes from 172.17.0.4: icmp_seq=1 ttl=64 time=0.123 ms
Okay, so the pod to pod connection is working. Keep in mind that container images are minimal, so you may install ping as I did in my example. Also, depending on your application you can use different methods for checking connectivity - I could use curl command as well and I will get the standard NGINX home page.
Now time to create services (I am assuming this is what you mean by microservice) and test connectivity. Service is an abstract mechanism for exposing pods on a network. So we can test connectivity either by getting list of endpoints - IP address of the pods associated with this service - kubectl get endpoints my-service, and then checking pod to pod connection like in previous example, or we can just curl service IP address/hostname. For hostname between namespaces it's little bit different! Check below:
Let's create deployments with 3 replicas:
kubectl create deployment nginx --image=nginx --replicas=3
kubectl create deployment nginx --image=nginx --replicas=3 -n test
For each deployment we will create service using kubectl expose:
kubectl expose deployment nginx --name=my-service --port=80
kubectl expose deployment nginx --name=my-service-test --port=80 -n test
Time to get IP addresses of the services:
user@shell:~$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 64d
my-service ClusterIP 10.107.224.54 <none> 80/TCP 12m
And in test namespace:
user@shell:~$ kubectl get svc -n test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-service-test ClusterIP 10.110.51.62 <none> 80/TCP 8s
I will exec into pod in default namespace and curl IP address of the my-service-test in second namespace:
user@shell:~$ kubectl exec -it nginx-6799fc88d8-w5q8s -- sh
# curl 10.110.51.62
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
Okay, it's working... Let's try with hostname:
# curl my-service-test
curl: (6) Could not resolve host: my-service-test
Not working... why? Let's check /etc/resolv.conf file:
# cat resolv.conf
nameserver 10.96.0.10
search test.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
It's looking for hostnames only in namespace where pod is located.
So pod in the test namespace will have something like:
# cat resolv.conf
nameserver 10.96.0.10
search test.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
Let's try to curl my-service-test.test.svc.cluster.local from pod in default namespace:
# curl my-service-test.test.svc.cluster.local
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
It's working.
To sum up:
- communication between pods in cluster should work properly between all namespaces (assuming that you have proper CNI plugin installed, on AWS EKS you have)
- pods containers are in most cases Linux containers so just use Linux tools to check connectivity (like
pingorcurl) - Services are an abstract mechanism for exposing pods on a network, you can connect to them for example using
curlcommand - IP addresses are cluster wide, hostnames are namespace wide - if you want to connect to the resource from the other namespace you need to use fully-qualified name (it applies for services, pods...)
Also check these articles:
Related