Cert-Manager k8s, Renewing certificate process

user_1958107 0
Viewed 2945

I have installed cert manager on a k8s cluster:

helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.5.3 --set installCRDs=true

My objective is to do mtls communication between micro-services running in same name-space.

For this purpose I have created a ca issuer .i.e..

kubectl get issuer -n sandbox -o yaml
apiVersion: v1
items:
- apiVersion: cert-manager.io/v1
  kind: Issuer
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"cert-manager.io/v1","kind":"Issuer","metadata":{"annotations":{},"name":"ca-issuer","namespace":"sandbox"},"spec":{"ca":{"secretName":"tls-internal-ca"}}}
    creationTimestamp: "2021-09-16T17:24:58Z"
    generation: 1
    managedFields:
    - apiVersion: cert-manager.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:spec:
          .: {}
          f:ca:
            .: {}
            f:secretName: {}
      manager: HashiCorp
      operation: Update
      time: "2021-09-16T17:24:58Z"
    - apiVersion: cert-manager.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:conditions: {}
      manager: controller
      operation: Update
      time: "2021-09-16T17:24:58Z"
    name: ca-issuer
    namespace: sandbox
    resourceVersion: "3895820"
    selfLink: /apis/cert-manager.io/v1/namespaces/sandbox/issuers/ca-issuer
    uid: 90f0c811-b78d-4346-bb57-68bf607ee468
  spec:
    ca:
      secretName: tls-internal-ca
  status:
    conditions:
      message: Signing CA verified
      observedGeneration: 1
      reason: KeyPairVerified
      status: "True"
      type: Ready
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Using this ca issuer, I have created certificates for my two micro-service .i.e.

kubectl get certificate -n sandbox      
NAME                       READY         SECRET             Age
service1-certificate        True    service1-certificate     3d     
service2-certificate        True    service2-certificate     2d23h

which is configured as

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  annotations:
    meta.helm.sh/release-name: service1
    meta.helm.sh/release-namespace: sandbox
  creationTimestamp: "2021-09-17T10:20:21Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  managedFields:
  - apiVersion: cert-manager.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:meta.helm.sh/release-name: {}
          f:meta.helm.sh/release-namespace: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        .: {}
        f:commonName: {}
        f:dnsNames: {}
        f:duration: {}
        f:issuerRef:
          .: {}
          f:kind: {}
          f:name: {}
        f:renewBefore: {}
        f:secretName: {}
        f:subject:
          .: {}
          f:organizations: {}
        f:usages: {}
    manager: Go-http-client
    operation: Update
    time: "2021-09-17T10:20:21Z"
  - apiVersion: cert-manager.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:privateKey: {}
      f:status:
        .: {}
        f:conditions: {}
        f:notAfter: {}
        f:notBefore: {}
        f:renewalTime: {}
        f:revision: {}
    manager: controller
    operation: Update
    time: "2021-09-20T05:14:12Z"
  name: service1-certificate
  namespace: sandbox
  resourceVersion: "5177051"
  selfLink: /apis/cert-manager.io/v1/namespaces/sandbox/certificates/service1-certificate
  uid: 0cf1ea65-92a1-4b03-944e-b847de2c80d9
spec:
  commonName: example.com
  dnsNames:
  - service1
  duration: 24h0m0s
  issuerRef:
    kind: Issuer
    name: ca-issuer
  renewBefore: 12h0m0s
  secretName: service1-certificate
  subject:
    organizations:
    - myorg
  usages:
  - client auth
  - server auth
status:
  conditions:
  - lastTransitionTime: "2021-09-20T05:14:13Z"
    message: Certificate is up to date and has not expired
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready
  notAfter: "2021-09-21T05:14:13Z"
  notBefore: "2021-09-20T05:14:13Z"
  renewalTime: "2021-09-20T17:14:13Z"
  revision: 5

Now as you could see in the configuration I have configured to renew them in 12 hours. However, the secrets created via this custom certificate resource are still aged to two days (the first it was created). I was thinking this tls secret will be renewed via cert manager each day) .i.e.

kubectl get secrets service1-certificate service2-certificate -n sandbox -o wide
NAME                        TYPE                DATA   AGE
service1-certificate        kubernetes.io/tls   3      2d23h
service2-certificate        kubernetes.io/tls   3      3d1h

Is there is some wrong in my understanding? In the certmangager pod logs I do see some error around renewing .i.e.

I0920 05:14:04.649158       1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key”=“sandbox/service1-certificate" "message"="Renewing certificate as renewal was scheduled at 2021-09-19 08:24:13 +0000 UTC" "reason"="Renewing"
    
I0920 05:14:04.649235       1 conditions.go:201] Setting lastTransitionTime for Certificate “service1-certificate" condition "Issuing" to 2021-09-20 05:14:04.649227766 +0000 UTC m=+87949.327215532
    
I0920 05:14:04.652174       1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key"="sandbox/service2 "message"="Renewing certificate as renewal was scheduled at 2021-09-19 10:20:22 +0000 UTC" "reason"="Renewing"
    
I0920 05:14:04.652231       1 conditions.go:201] Setting lastTransitionTime for Certificate “service2-certificate" condition "Issuing" to 2021-09-20 05:14:04.652224302 +0000 UTC m=+87949.330212052
    
I0920 05:14:04.671111       1 conditions.go:190] Found status change for Certificate “service2-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:04.671094596 +0000 UTC m=+87949.349082328
I0920 05:14:04.671344       1 conditions.go:190] Found status change for Certificate “service1-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:04.671332206 +0000 UTC m=+87949.349319948
    
    
I0920 05:14:12.703039       1 controller.go:161] cert-manager/controller/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox/service2-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service2-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
    
I0920 05:14:12.703896       1 conditions.go:190] Found status change for Certificate “service2-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:12.7038803 +0000 UTC m=+87957.381868045
    
    
I0920 05:14:12.749502       1 controller.go:161] cert-manager/controller/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox/service1-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service1-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
    
I0920 05:14:12.750096       1 conditions.go:190] Found status change for Certificate “service1-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:12.750082572 +0000 UTC m=+87957.428070303
    
I0920 05:14:13.009032       1 controller.go:161] cert-manager/controller/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key"="sandbox/service1-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service1-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
I0920 05:14:13.117843       1 controller.go:161] cert-manager/controller/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox/service2-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service2-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
I0920 05:14:13.119366       1 conditions.go:190] Found status change for Certificate “service2-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:13.119351795 +0000 UTC m=+87957.797339520
    
I0920 05:14:13.122820       1 controller.go:161] cert-manager/controller/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox\service2-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
I0920 05:14:13.123907       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “service2-certificate-t92qh" condition "Approved" to 2021-09-20 05:14:13.123896104 +0000 UTC m=+87957.801883833
I0920 05:14:13.248082       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “service1-certificate-p9stz" condition "Approved" to 2021-09-20 05:14:13.248071551 +0000 UTC m=+87957.926059296
I0920 05:14:13.253488       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “serivce2-certificate-t92qh" condition "Ready" to 2021-09-20 05:14:13.253474153 +0000 UTC m=+87957.931461871
I0920 05:14:13.388001       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “service1-certificate-p9stz" condition "Ready" to 2021-09-20 05:14:13.387983783 +0000 UTC m=+87958.065971525

cert-manager kubernetes
1 Answers

Short answer

Based on logs and details from certificate you provided it's safe to say it's working as expected.

Pay attention to revision: 5 in your certificate, which means that certificate has been renewed 4 times already. If you try to look there now, this will be 6 or 7 because certificate is updated every 12 hours.

Logs

First thing which can be really confusing is error messages in cert-manager pod. This is mostly noisy messages which are not really helpful by itself.

See about it here Github issue comment and here github issue 3667.

In case logs are really needed, verbosity level should be increased by setting args to --v=5 in the cert-manager deployment. To edit a deployment run following command:

kubectl edit deploy cert-manager -n cert-manager

How to check certificate/secret

When certificate is renewed, secret's and certificate's age are not changed, but content is edited, for instance resourceVersion in secret and revision in certificate.

Below are options to check if certificate was renewed:

  1. Check this by getting secret in yaml before and after renew:

    kubectl get secret example-certificate -o yaml > secret-before

And then run diff between them. It will be seen that tls.crt as well as resourceVersion is updated.

  1. Look into certificate revision and dates in status (I set duration to minimum possible 1h and renewBefore 55m, so it's updated every 5 minutes):

     $ kubectl get cert example-cert -o yaml
 notAfter: "2021-09-21T14:05:24Z"
 notBefore: "2021-09-21T13:05:24Z"
 renewalTime: "2021-09-21T13:10:24Z"
 revision: 7

  2. Check events in the namespace where certificate/secret are deployed:

    $ kubectl get events
117s        Normal   Issuing             certificate/example-cert                The certificate has been successfully issued
117s        Normal   Reused              certificate/example-cert                Reusing private key stored in existing Secret resource "example-staging-certificate"
6m57s       Normal   Issuing             certificate/example-cert                Renewing certificate as renewal was scheduled at 2021-09-21 13:00:24 +0000 UTC
6m57s       Normal   Requested           certificate/example-cert                Created new CertificateRequest resource "example-cert-bs8g6"
117s        Normal   Issuing             certificate/example-cert                Renewing certificate as renewal was scheduled at 2021-09-21 13:05:24 +0000 UTC
117s        Normal   Requested           certificate/example-cert                Created new CertificateRequest resource "example-cert-7x8cf" UTC

  3. Look at certificaterequests:

    $ kubectl get certificaterequests
NAME                 APPROVED   DENIED   READY   ISSUER      REQUESTOR                                         AGE
example-cert-2pxdd   True                True    ca-issuer   system:serviceaccount:cert-manager:cert-manager   14m
example-cert-54zzc   True                True    ca-issuer      system:serviceaccount:cert-manager:cert-manager   4m29s
example-cert-8vjcm   True                True    ca-issuer      system:serviceaccount:cert-manager:cert-manager   9m29s

  4. Check logs in cert-manager pod to see four stages:

    I0921 12:45:24.000726       1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key"="default/example-cert" "message"="Renewing certificate as renewal was scheduled at 2021-09-21 12:45:24 +0000 UTC" "reason"="Renewing"
I0921 12:45:24.000761       1 conditions.go:201] Setting lastTransitionTime for Certificate "example-cert" condition "Issuing" to 2021-09-21 12:45:24.000756621 +0000 UTC m=+72341.194879378
I0921 12:45:24.120503       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "example-cert-mxvbm" condition "Approved" to 2021-09-21 12:45:24.12049391 +0000 UTC m=+72341.314616684
I0921 12:45:24.154092       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "example-cert-mxvbm" condition "Ready" to 2021-09-21 12:45:24.154081971 +0000 UTC m=+72341.348204734

Note

Very important that not all issuers support duration and renewBefore flags. E.g. letsencrypt still doesn't work with it and have 90 default days.

Refence.

user_155372010
Related