I used the kubectl with yaml files to create target pod, and found the process in the pod is listening target port as expected.
To my surprise, the Port is not seen in the outputs of netstat -tunlp or netstat -alp or netstat -an from host machine. But it works if I try telnet localhost targetPort!!!
Why this happens?? Can anybody explain this?
This is because of docker. By default docker does not add container network namespaces to the linux runtime data (/var/run mounted as a tmpfs from /run) which is what you see when you run the ip netns command.
To view the network namespaces you need to use nsenter.
docker ps
docker inspect --format '{{ .State.Pid }}' <<container-id>>
nsenter over docker exec is that nsenter will enable you to execute all tools or commands available on the node inside the pod where as docker exec will allow only limited or restricted commands.nsenter -t <<container pid>> -n netstat -tunlp
In order to access the service on a worker node, you have to expose the pod on the worker node with a service of type NodePort. Processes running inside the pods are in a different networking namespace. You can access the application from inside the pod but not from the node without the service object. See the following for ref:
$ kubectl create deploy nginx --image=nginx
$ kubectl expose deploy nginx --target-port 80 --port 80 --type NodePort
$ NODE_PORT=$(kubectl get svc nginx -ojsonpath='{ .spec.ports[0].nodePort }')
$ netstat -an | grep $NODE_PORT
tcp46 0 0 *.31563 *.* LISTEN
$ curl localhost:$NODE_PORT
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
HTH
Since containers inside pods are running in their own network namespace therefore netstat is unable to detect them.
For listing ports listening inside containers use nsenter. This tool will help you to execute command on a different namespace of a proces (in our case PID of desired container).
docker inspect <containerid> (if docker is your container runtime)and, after grabbing PID from above command run
$ nsenter -t <container-PID> -n netstat -nltpnsenter -t <container-PID> -n netstat -nltp
this command show container port, not host port, still now answer the question
Incoming connections might directly be routed into the k8s network using iptable rules (e.g. when using Calico CNI).
You can try
iptable -L -t nat
and see if your hostports turn up there.
See the explanation here:
https://www.reddit.com/r/kubernetes/comments/kne734/help_demystify_hostport_networking_please/