Prometheus pod crashlooping on error "opening storage failed"

Question

I restarted the Prometheus pod and now the Prometheus pod is crashlooping. Found this error in the logs:

level=error ts=2021-06-09T09:27:29.066Z caller=main.go:758 err="opening storage failed: block dir: \"/prometheus/01F6J0P4KBBWVJD2M8B1PE7C5E\": open /prometheus/01F6J0P4KBBWVJD2M8B1PE7C5E/meta.json: no such file or directory"

Noticed that the 01F6J0P4KBBWVJD2M8B1PE7C5E folder only has the chunks folder in it.

Any idea why this occurs and is there a way to fix this?

user_9929015 · Answer

So you have already found correct github related issues and asked there also.

And seems currently the only way to fix the problem is to delete 01F6J0P4KBBWVJD2M8B1PE7C5E folder that contains empty meta.json either nothing (link)

Prometheus pod crashlooping on error "opening storage failed"

1 Answers