I have problem with networking to one single container in Pod. I have created kubernetes cluster (one master and one node) for learning. Master is installed on one VM and Node is installed on second one VM.
Node listing:
NAME STATUS ROLES AGE VERSION
kmaster Ready control-plane,master 68m v1.20.2
ubuntu Ready <none> 57m v1.20.2
Cluster info:
Kubernetes control plane is running at https://192.168.0.102:6443
KubeDNS is running at https://192.168.0.102:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Everything was working fine until VM's restarting.
One Pod with eclipse-mosquitto mqtt broker has some problem with networking with container inside.
Below listing of pods:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mqttbroker-574b696f76-x26jq 1/1 Running 0 57m 192.168.243.193 ubuntu <none> <none>
Below description of problematic pod:
Name: mqttbroker-574b696f76-x26jq
Namespace: default
Priority: 0
Node: ubuntu/192.168.0.121
Start Time: Mon, 08 Feb 2021 00:18:16 -0800
Labels: app=mqttbroker
pod-template-hash=574b696f76
Annotations: cni.projectcalico.org/podIP: 192.168.243.193/32
cni.projectcalico.org/podIPs: 192.168.243.193/32
Status: Running
IP: 192.168.243.193
IPs:
IP: 192.168.243.193
Controlled By: ReplicaSet/mqttbroker-574b696f76
Containers:
mosquitto:
Container ID: docker://73417f8046abfd8773154cde02b7366f9d7c26331b03b141e89855deec71e273
Image: eclipse-mosquitto
Image ID: docker-pullable://eclipse-mosquitto@sha256:0916ce445d7f3945168966ab987515a081f2f2c761acb85ae00ff44d97d3a84f
Port: 1883/TCP
Host Port: 0/TCP
State: Running
Started: Mon, 08 Feb 2021 00:18:20 -0800
Ready: True
Restart Count: 0
Limits:
cpu: 750m
memory: 700Mi
Requests:
cpu: 500m
memory: 500Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-qgs7v (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-qgs7v:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-qgs7v
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
But when I inspect logs of container, I get some error with assigning IP: Container logs
1612772300: mosquitto version 2.0.7 starting
1612772300: Config loaded from /mosquitto/config/mosquitto.conf.
1612772300: Starting in local only mode. Connections will only be possible from clients running on this machine.
1612772300: Create a configuration file which defines a listener to allow remote access.
1612772300: Opening ipv4 listen socket on port 1883.
1612772300: Opening ipv6 listen socket on port 1883.
1612772300: Error: Address not available
1612772300: mosquitto version 2.0.7 running
When I have checked addresses assigned to container, I saw strange IP mask. I am supposing that this could be my problem. Below ifconfig output.
eth0 Link encap:Ethernet HWaddr 5A:27:37:13:F7:06
inet addr:192.168.243.193 Bcast:192.168.243.193 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:14328 errors:0 dropped:0 overruns:0 frame:0
TX packets:14315 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:927289 (905.5 KiB) TX bytes:773046 (754.9 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:138 errors:0 dropped:0 overruns:0 frame:0
TX packets:138 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7597 (7.4 KiB) TX bytes:7597 (7.4 KiB)
Below deployment and services:
kind: Deployment
apiVersion: apps/v1
metadata:
name: mqttbroker
labels:
app: mqttbroker
spec:
replicas: 1
selector:
matchLabels:
app: mqttbroker
template:
metadata:
labels:
app: mqttbroker
spec:
containers:
- name: mosquitto
image: eclipse-mosquitto
resources:
requests:
cpu: "0.5"
memory: "500Mi"
limits:
cpu: "0.75"
memory: "700Mi"
ports:
- name: mqttbroker-1883
containerPort: 1883
Cluster IP services
apiVersion: v1
kind: Service
metadata:
name: mosquitto-service
spec:
type: ClusterIP
selector:
app: mqttbroker
ports:
- name: service-mqtt-1883
protocol: TCP
port: 1883
targetPort: 1883
Nodeport IP services
apiVersion: v1
kind: Service
metadata:
name: mosquittoservice
spec:
type: NodePort
ports:
- name: "1883"
port: 80
targetPort:
nodePort: 30081 # acces service via external port number
selector:
app: mqttbroker
Listing of services:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 116m <none>
mosquitto-service ClusterIP 10.98.70.62 <none> 1883/TCP 101m app=mqttbroker
mosquittoservice NodePort 10.106.207.192 <none> 80:30081/TCP 101m app=mqttbroker
Description of mosquitto-service
Name: mosquitto-service
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=mqttbroker
Type: ClusterIP
IP Families: <none>
IP: 10.98.70.62
IPs: 10.98.70.62
Port: service-mqtt-1883 1883/TCP
TargetPort: 1883/TCP
Endpoints: 192.168.243.193:1883
Session Affinity: None
Events: <none>
Description of mosquittoservice
Name: mosquittoservice
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=mqttbroker
Type: NodePort
IP Families: <none>
IP: 10.106.207.192
IPs: 10.106.207.192
Port: 1883 80/TCP
TargetPort: 1883/TCP
NodePort: 1883 30081/TCP
Endpoints: 192.168.243.193:1883
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
My question is what could cause this problem and how to fix it?
Thank you for answer!!
Edit: When I am invoking container's function like below is working fine.
kubectl exec -it mqttbroker-574b696f76-x26jq -- mositto_sub -h localhost -p 1883 -t topic -C 4
If I change "localhost" to IP adress, I get connection refused.
kubectl exec -it mqttbroker-574b696f76-x26jq -- mositto_sub -h 192.168.0.102 -p 1883 -t topic -C 4
Error: Connection refused
command terminated with exit code 1
If I invoked function like below, I get also connection refused.
kubectl exec -it mqttbroker-574b696f76-x26jq -- mosquitto_sub -h 192.168.0.102 -p 30081 -t topic -C 4
Error: Connection refused
command terminated with exit code 1