"Connect: Connection Refused" when Connecting Prometheus to Kubernetes

user_10251749 0
Viewed 9787

I am new to Prometheus and relatively new to kubernetes so bear with me, please. I am trying to test Prometheus out and have tried two different approaches.

  1. Run Prometheus as a docker container outside of kubernetes. To accomplish this I have created this Dockerfile:

    FROM prom/prometheus
ADD prometheus.yml /etc/prometheus/

    and this yaml file:

    global:
  scrape_interval: 15s
  external_labels:
    monitor: 'codelab-monitor'
scrape_configs:
- job_name: 'kubernetes-apiservers'
  scheme: http
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  - role: endpoints
    api_server: localhost:443

    When I run this I get:

    Failed to list *v1.Pod: Get http://localhost:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: connect: connection refused"
Failed to list *v1.Service: Get http://localhost:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: connect: connection refused"
Failed to list *v1.Endpoints: Get http://localhost:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: connect: connection refused"

    on a loop. Prometheus will load when I go to localhost:9090 but there is no data.

  2. I thought deploying Prometheus as a Kubernetes deployment may help, so I made this yaml and deployed it.

    kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: prometheus-monitor
spec:
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus-monitor
        image: prom/prometheus
        # args:
        #   - '-config.file=/etc/prometheus/prometheus.yaml'
        imagePullPolicy: IfNotPresent
        ports:
        - name: webui
          containerPort: 9090

    The deployment was successful, but if I go to localhost:9090 I get 'ERR_SOCKET_NOT_CONNECTED'. (my port is forwarded)

Can anyone tell me the advantage of in vs out of Kubernetes and how to fix at least one of these issues?

Also, my config file is suppressed because it was giving an error, and I will look into that once I am able to get Prometheus loaded.

prometheus kubernetes docker yaml
2 Answers

Kubernetes does not map the port outside it's cluster when you deploy your container.

You also have to create a service (can be inside the same file) to make it available from your workstation (append this to your prometheus yaml):

---
apiVersion: v1
kind: Service
metadata:
    name: prometheus-web
    labels:
        app: prometheus
spec:
    type: NodePort
    ports:
        - port: 9090
          protocol: TCP
          targetPort: 9090
          nodePort: 30090
          name: webui
    selector:
        app: prometheus

NodePort opens the given port on all nodes you have. You should be able to see the frontend with http://localhost:30090/

Per default, kubernetes allow ports 30000 to 32767 for NodePort type (https://kubernetes.io/docs/concepts/services-networking/service/#nodeport).

Please consider reading the documentation in general for more information on services in kubernetes: https://kubernetes.io/docs/concepts/services-networking/service/

user_20872600

So 2 different issues. On:

  1. You are trying to connect to localhost:443 where Prometheus is running and it's expecting to talk to a Kubernetes API server. Apparently, nothing is listening on localhost:443. Are you doing port forwarding to your kube-apiserver?

  2. In this case you need to expose your deployment port. With something like:

     kubectl expose deployment prmetheus-web --type=LoadBalancer # or 
 kubectl expose deployment prmetheus-web --type=NodePort

    depending on how you want to expose your service. NodePort exposes it in service that maps to a port on your Kubernetes nodes (IPAddress:Port) and LoadBalancer exposes the deployment using an external load balancer that may vary depending on what cloud you are using (AWS, GCP, OpenStack, Azure, etc). More about exposing your Deployments or DaemonSets or StatefulSets here. More about services here

Hope it helps.

user_29892610
Related