kubernetes redinessProbe httpGet on tomcat image reports connection refused

user_3192775 0
Viewed 967

I am using an initContainer to mount a index.html and mount that within the tomcat 9 server. The readinessProbe, type: httpGet with initalDelaySeconds: 60.

Without the readinessProbe, the application started and i am able to access the Url and see the content.

kubectl describe pod/<pod--> provides below message

 Normal   Created    104s               kubelet, worker1.com  Created container tomcat-container
 Normal   Started    104s               kubelet, worker1.com  Started container tomcat-container
 Warning  Unhealthy  10s (x4 over 40s)  kubelet, worker1.com  Readiness probe failed: Get http://localhost:8080/temp/index.html: dial tcp [::1]:8080: connect: connection refused

without the readiness probe when i exec into the container and perform curl command i see the responce.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
     app: tomcat
  template:
    metadata:
      labels:
        app: tomcat
        role: rolling-update
    spec:
      volumes:
      - name: shared-volume
        emptyDir: {}
      initContainers:
      - name: busybox
        image: busybox
        volumeMounts:
        - name: shared-volume
          mountPath: /app-data/temp
        command: ["/bin/sh"]
        args: ["-c","echo '<h1>hellow from k8s!</h1>' > /app-data/temp/index.html"]

      containers:
      - name: tomcat-container
        image: tomcat:9.0
        # command: ["/bin/bash"]
        # args: ["-c","mkdir -p /usr/local/tomcat/webapps/temp; touch index.html"]
        volumeMounts:
        - name: shared-volume
          mountPath: /usr/local/tomcat/webapps/temp
        ports:
        - containerPort: 8080
        readinessProbe:
          initialDelaySeconds: 60
          httpGet:
            host: localhost
            path: /temp/index.html
            port: 8080
        #  exec:
        #    command:
        #      - curl
        #      - http://localhost:8080/temp/index.html
  strategy:
    type: RollingUpdate
    rollingUpdate:
       maxSurge: 2
       maxUnavailable: 1
---
kind: Service
apiVersion: v1
metadata:
  name: tomcat-service
spec:
  selector:
    app: tomcat
  type: NodePort
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
    nodePort: 30080

The readinessProbe exec: command works though, the pod is started successfully.

Question: Any inputs why the httpGet readiness didn't work in this case.

Logs from the container:

27-Jun-2020 22:41:17.709 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
27-Jun-2020 22:41:17.710 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/local/tomcat
27-Jun-2020 22:41:17.710 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat
27-Jun-2020 22:41:17.711 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
27-Jun-2020 22:41:17.711 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.24] using APR version [1.6.5].
27-Jun-2020 22:41:17.711 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
27-Jun-2020 22:41:17.712 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
27-Jun-2020 22:41:17.747 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1d  10 Sep 2019]
27-Jun-2020 22:41:23.913 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
27-Jun-2020 22:41:24.324 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [9,608] milliseconds
27-Jun-2020 22:41:25.112 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
27-Jun-2020 22:41:25.119 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.36]
27-Jun-2020 22:41:25.194 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/temp]
27-Jun-2020 22:41:27.980 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/temp] has finished in [2,766] ms
27-Jun-2020 22:41:28.100 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
27-Jun-2020 22:41:28.286 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [3,960] milliseconds
kubernetes tomcat
2 Answers

Exec works because that's being executed within the container where localhost:8080 is available. But httpGet is executed by Kubelet. So when host is specified as localhost, it tries to connect to port 8080 on the node. Hence the failure.

Please remove "host: localhost" in probe so that it'll default to pod ip and will work as expected.

user_67763540

when i was developing the deployment, i used the /manager/html as the path for httpGet probe. Which was not working due to 404 error.

To fix it, i tried out few option including host localhost, after changing the path i didn't remove the localhost. As mentioned by @Naga venkata Pradeep Namburi.

Kubelet uses the container ip for httpGet, specifying host : localhost it assumes the localhost of node rather than the container itself.

  readinessProbe:
          initialDelaySeconds: 60
          httpGet:
           # host: localhost  # -- remove this local host
            path: /temp/index.html
            port: 8080
user_31927750
Related