Prometheus kuberentes-pods Get "https:// xx.xx.xx:443 /metrics": dial tcp xx.xx.xx:443: connect: connection refused

user_14096639 0
Viewed 327

I have configured Prometheus on one of the kubernetes cluster nodes using [this][1]. After that I added following prometheus.yml file. I can list nodes and apiservers but for pods, all the pods shows down and error:

Get "https:// xx.xx.xx:443 /metrics": dial tcp xx.xx.xx:443: connect: connection refused and for some pods the status is unknown.

Can someone point me what am I doing wrong here?

Cat prometheus.yml



global:
  scrape_interval: 1m
 
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: \['localhost:9090'\]
 
# metrics for default/kubernetes api's from the kubernetes master
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
      bearer_token_file: /dfgdjk/token
      api_server: https://masterapi.com:3343
      tls_config:
        insecure_skip_verify: true
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /dfgdjk/token
    scheme: https
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: \[__meta_kubernetes_namespace\]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: \[__meta_kubernetes_pod_name\]
      action: replace
      target_label: kubernetes_pod_name
 
# metrics for default/kubernetes api's from the kubernetes master
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
    - role: endpoints
      api_server: https://masterapi.com:3343
      bearer_token_file: /dfgdjk/token
      tls_config:
        insecure_skip_verify: true
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /dfgdjk/token
    scheme: https
    relabel_configs:
    - source_labels: \[__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name\]
      action: keep
      regex: default;kubernetes;https][1]
 


  [1]: https://devopscube.com/install-configure-prometheus-linux/
prometheus kubernetes-pod kubernetes
1 Answers

It's impossible to get metrics to external prometheus server without having any prometheus components inside the kubernetes cluster. This happens because cluster network is isolated with host's network and it's not possible to scrape metrics from pods directly from outside the cluster.

Please refer to Monitoring kubernetes with prometheus from outside of k8s cluster GitHub issue

There options which can be done:

  • install prometheus inside the cluster using prometheus operator or manually - example
  • use proxy solutions, for example this one from the same thread on GitHub - k8s-prometheus-proxy
  • on top of the prometheus installed within the cluster, it's possible to have external prometheus in federation so all logs are saved outside of the cluster. Please refer to prometheus federation.

Also important part is kube state metrics should be installed as well in kubernetes cluster. How to set it up.

Edit: also you can refer to another SO question/answer which confirms that only with additional steps or OP resolved it by another proxy solution.

user_155372010
Related