How to get MinIO metrics (/minio/v2/metrics/cluster)?

user_4436022 0
Viewed 2229

Bases on the MinIO documentation at page 14 following API is mentioned /minio/v2/metrics/cluster and /minio/v2/metrics/node.
But looks like these API available only for "Prometheus". And I have no clue to get those using python.

Of course if I try send request to http://<host>:<port>/minio/v2/metrics/cluster I get 403 error:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AllAccessDisabled</Code><Message>All access to this bucket has been disabled.</Message><Resource>/minio/v2/metrics/cluster</Resource><RequestId></RequestId><HostId><uuid-here></HostId></Error>

I've tried to generate presigned URL (like for GET object from bucket) but even with signature result is same as above.

Maybe someone know to reach that API(s)?
And how to do proper authentication for these endpoint without "Prometheus"?

minio boto3 python
2 Answers

Prometheus just requests an HTTP resource, which presents metrics in plain text format. You can grab the same URL that prometheus would use and then parse it yourself.

If you look at this document, it describes how to configure authentication for the metrics endpoint.

  1. Configure authentication type for Prometheus metrics

MinIO supports two authentication modes for Prometheus either jwt or public, by default MinIO runs in jwt mode. To allow public access without authentication for prometheus metrics set environment as follows.

export MINIO_PROMETHEUS_AUTH_TYPE="public" minio server ~/test

If you're running it in jwt mode (the default), the following sections tells you how to generate a configuration that will show you the appropriate bearer token to use for authentication:

3.1 Authenticated Prometheus config

If MinIO is configured to expose metrics without authentication, you don't need to use mc to generate prometheus config. You can skip reading further and move to 3.2 section.

The Prometheus endpoint in MinIO requires authentication by default. Prometheus supports a bearer token approach to authenticate prometheus scrape requests, override the default Prometheus config with the one generated using mc. To generate a Prometheus config for an alias, use mc as follows mc admin prometheus generate .

The command will generate the scrape_configs section of the prometheus.yml as follows:

scrape_configs:
- job_name: minio-job
  bearer_token: <secret>
  metrics_path: /minio/v2/metrics/cluster
  scheme: http
  static_configs:
  - targets: ['localhost:9000']

You can just use the bearer token and path from that configuration to access your metrics. The bearer token should go into an Authorization header (Authorization: Bearer <token>).

user_1473560

Solution:

import jwt  # pyJWT
import requests
import datetime

expired_at = (datetime.datetime.now() + datetime.timedelta(days=1)).timestamp()

token = jwt.encode(
    payload={
        "exp": expired_at,
        "sub": "<AccessKey>",
        "iss": "prometheus",
    },
    key="<SecretKey>",
    algorithm="HS512"
)

headers = {
    "Authorization": f"Bearer {token.decode()}"
}
response = requests.get(
    url="<schema>://<host>:<port>/minio/prometheus/metrics",
    headers=headers
)

How mc utility generates bearer token I found at https://github.com/minio/mc/blob/master/cmd/admin-prometheus-generate.go

URL /minio/prometheus/metrics was found during call to utility mc admin prometheus generate test-minio

user_44360220
Related