Where does Prometheus store metrics data (in container)?

user_13285647 0
Viewed 11034

I am searching for the ./data folder described in Storage section pf Prometheus Documentation. I run a basic Prometheus Docker container prom/prometheus on Kubernetes. If I execute a shell inside the container, the working directory is /prometheus and it contains the wal directory, but it does not have the structure mentioned in the documentation and I can not find any metric data.

Where are the metrics stored which I can query over Prometheus GUI?

prometheus docker
2 Answers

As you can check the Prometheus Dockerfile on Github (https://github.com/prometheus/prometheus/blob/master/Dockerfile#L24), the working directory is /prometheus and that where you will find all the metrics and data.

Below is the data present in /prometheus directory

/prometheus $ ls -l
total 8
-rw-r--r--    1 nobody   nogroup          0 May 28 12:39 lock
-rw-r--r--    1 nobody   nogroup      20001 May 28 12:45 queries.active
drwxr-xr-x    2 nobody   nogroup       4096 May 28 12:39 wal

And this is the *Time Series Database * which you can't decode . If you will run more exporters, probably you can see more folders like chunks etc.

Ref: https://prometheus.io/docs/prometheus/1.8/storage/

user_80901740

I have figured out the problem now. The structure, which was mentioned in the documentation, need several hours to build up.

./data
├── 01BKGTZQ1HHWHV8FBJXW1Y3W0K
│   └── meta.json
├── 01BKGV7JC0RY8A6MACW02A2PJD
│   ├── chunks
│   │   └── 000001
│   ├── tombstones
│   ├── index
│   └── meta.json
└── wal
    ├── 00000002
    └── checkpoint.000001

The first of those folders starting with 01... appeared after 4 hours. The next one took an other 2 hours. It seems to vary according to the amount of metrics you are pulling.

Sidenote: As nischay goyal wrote in his answer, Prometheus TSDB can not be decoded. You can query the Prometheus API with a script to export metrics.

user_132856470
Related