We would like to implement a monitoring system (Prometheus and Grafana), into our microservices architecture (we are using AWS), we are currently working with +2 ECS clusters with fargate, and have several EC2 instances running independently (databases services mainly), and lambdas for some tasks.
I have some questions regarding the implementation:
1. Should we install prometheus on each cluster to collect the data of the cluster and then, send it to a master prometheus to handle the data, and maybe bring in another tool like Thanos?
2. Or is it better to install exporters in the ec2 instances (node exporter, mysql exporter, and so on), and libraries in the services (Node APIS mainly, such as prom-client), and tasks running in AWS ECS cluster, and expose metrics data via HTTP Endpoints and consume it in a central Prometheus installation, something like this, (example of our oversimplied microservice architecture):
Do you have any kind of diagram showing a similar implementation in AWS? it will be very helpful. Thanks!