I have an application running on kubernetes (It is a cluster running on cloud) and want to setup monitoring and logging for that application. There are various possibilities for the setup. What would be the best practices of doing that, like recommended method or industrial standard?
- A prometheus monitoring setup inside kubernetes cluster: prometheus-operator helm chart installed inside the cluster that can monitor the entire cluster, including the application.
- an external prometheus + grafana setup deployed with docker-compose.(But I doubt if the external setup can reach the k8s properly to scrape all the metrics)
- A prometheus federation setup where one external prometheus setup gets metrics from an internal prometheus setup of k8s.
Can anyone please help me with some suggestions regarding best practices?