Architecture Overview:
We have Grafana running on Azure Kubernetes Service and Persistent Storage has been enabled.
We store the Grafana Helm value file in git and use ArgoCD for CD.
Now we are in a situation where we should store all dashboards and data sources config in the Helm values file or not.
Advantage:
- In case something is wrong with the persistent storage (e.g. disaster, someone deletes all Grafana resources), we can easily re-create Grafana resources.
Disadvantages:
- If a user would like to modify a data source, he/she has to notify the Grafana admin to update the Helm value file, then have a normal code review process to apply the change. So the users have to wait for a certain amount of time for a tiny little change.
- Later we may have more Grafana users, they may just create new data sources or dashboards as they wish. We just cannot source control everything.
I was wondering is there a common best practice for this situation? Or it really depends on our own situation?
Thanks!