Istio install with updated configs doesn't delete Prometheus ServiceMonitor objects

user_1204824 0
Viewed 762

I have Istio (version 1.16.3) configured with an external Prometheus and I have the Prometheus ServiceMonitor objects configured using the built in Prometheus operator based on the discussion in this issue: https://github.com/istio/istio/issues/21187

For most part this works fine, except that I noticed that the kubernetes-services-secure-monitor and the kubernetes-pods-secure-monitor were also created and this resulted in Prometheus throwing certificate not found errors, as expected because I have not set these up.

"level=error ts=2020-07-06T03:43:33.464Z caller=manager.go:188 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /etc/prometheus/secrets/istio.prometheus/root-cert.pem: open /etc/prometheus/secrets/istio.prometheus/root-cert.pem: no such file or directory" scrape_pool=istio-system/kubernetes-pods-secure-monitor/0

I also noticed that the service monitor creation can be disabled by using the Values.prometheus.provisionPrometheusCert flag as per this:

istio/manifests/charts/istio-telemetry/prometheusOperator/templates/servicemonitors.yaml

 {{- if .Values.prometheus.provisionPrometheusCert }}

However, re-applying the config using `istioctl install did not delete those service monitors.

Does istioctl install command not delete/prune existing resources?

Here is my full configuration:

apiVersion: install.istio.io/v1alpha1
kind: IstioControlPlane
metadata:
  namespace: istio-system
  name: istio-controlplane
  labels:
    istio-injection: enabled
spec:
  profile: default
  addonComponents:
    prometheus:
      enabled: false
    prometheusOperator:
      enabled: true
    grafana:
      enabled: false
    kiali:
      enabled: true
      namespace: staging
    tracing:
      enabled: false
  values:
    global:
      proxy:
        logLevel: warning
      mountMtlsCerts: false
      prometheusNamespace: monitoring
      tracer:
        zipkin:
          address: jaeger-collector.staging:9411

    prometheusOperator:
      createPrometheusResource: false

    prometheus:
      security:
        enabled: false
      provisionPrometheusCert: false
prometheus-operator istio
1 Answers

Two separate concerns: Upgrade to a new version of Istio and updates to the config.

Upgrade

As far as I know there we´re a lot of issues when upgrading istio from older versions to 1.4,1.5,1.6, but recently when istioctl upgrade came up you shouldn´t be worried about upgrading your cluster.

The istioctl upgrade command performs an upgrade of Istio. Before performing the upgrade, it checks that the Istio installation meets the upgrade eligibility criteria. Also, it alerts the user if it detects any changes in the profile default values between Istio versions.

Additionally Istio 1.6 will support a new upgrade model to safely canary-deploy new versions of Istio. In this new model, proxies will associate with a specific control plane that they use. This allows a new version to deploy to the cluster with less risk - no proxies connect to the new version until the user explicitly chooses to. This allows gradually migrating workloads to the new control plane, while monitoring changes using Istio telemetry to investigate any issues

Related documentation about that is here and here.

Update

As I mentioned in comments, the 2 things I found which might help are

  • istioctl operator logs

If something with your update goes wrong then it will appear in istio operator logs, and the update will fail.

You can observe the changes that the controller makes in the cluster in response to IstioOperator CR updates by checking the operator controller logs:

$ kubectl logs -f -n istio-operator $(kubectl get pods -n istio-operator -lname=istio-operator -o jsonpath='{.items[0].metadata.name}')

Verify a successful installation You can check if the Istio installation succeeded using the verify-install command which compares the installation on your cluster to a manifest you specify.

If you didn’t generate your manifest prior to deployment, run the following command to generate it now:

$ istioctl manifest generate <your original installation options> > $HOME/generated-manifest.yaml

Then run the following verify-install command to see if the installation was successful:

$ istioctl verify-install -f $HOME/generated-manifest.yaml

Hope you find this useful.

user_119777600
Related