Issue with Self-signed certificate with Cert-Manager in Kubernetes

Question

I'm trying to add a self-signed certificate in my AKS cluster using Cert-Manager.

I created a ClusterIssuer for the CA certificate (to sign the certificate) and a second ClusterIssuer for the Certificate (self-signed) I want to use.

I am not sure if the certificate2 is being used correctly by Ingress as it looks like it is waiting for some event.

Am I following the correct way to do this?

This is the first ClusterIssuer "clusterissuer.yml":

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

This is the CA certificate "certificate.yml":

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: selfsigned-certificate
spec:
  secretName: hello-deployment-tls-ca-key-pair
  dnsNames:
  - "*.default.svc.cluster.local"
  - "*.default.com"
  isCA: true
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer

This is the second ClusterIssuer "clusterissuer2.yml" for the certificate I want to use:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
 name: hello-deployment-tls
spec:
 ca:
   secretName: hello-deployment-tls-ca-key-pair

and finally this is the self-signed certificate "certificate2.yml":

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: selfsigned-certificate2
spec:
  secretName: hello-deployment-tls-ca-key-pair2
  dnsNames:
  - "*.default.svc.cluster.local"
  - "*.default.com"
  isCA: false
  issuerRef:
    name: hello-deployment-tls
    kind: ClusterIssuer

I am using this certificate in an Ingress:

--- 
apiVersion: extensions/v1beta1
kind: Ingress
metadata: 
  annotations: 
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: "hello-deployment-tls"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  name: sonar-ingress
spec: 
  tls: 
  - secretName: "hello-deployment-tls-ca-key-pair2"
  rules: 
  - http: 
      paths: 
      - pathType: Prefix
        path: "/"
        backend: 
          serviceName: sonarqube
          servicePort: 80

As I do not have any registered domain name I just want to use the public IP to access the service over https://.

When I access to the service https:// I can see that "Kubernetes Ingress Controller Fake Certificate" so i guess this is because the certificate is not globally recognize by the browser.

The strange thing is here. Theoretically the Ingress deployment is using the selfsigned-certificate2 but looks like it is not ready:

kubectl get certificate
NAME                      READY   SECRET                              AGE
selfsigned-certificate    True    hello-deployment-tls-ca-key-pair    4h29m
selfsigned-certificate2   False   hello-deployment-tls-ca-key-pair2   3h3m
selfsigned-secret         True    selfsigned-secret                   5h25m

kubectl describe certificate selfsigned-certificate2
. 
.
.
Spec:
  Dns Names:
    *.default.svc.cluster.local
    *.default.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       hello-deployment-tls
  Secret Name:  hello-deployment-tls-ca-key-pair2
Status:
  Conditions:
    Last Transition Time:  2021-10-15T11:16:15Z
    Message:               Waiting for CertificateRequest "selfsigned-certificate2-3983093525" to complete
    Reason:                InProgress
    Status:                False
    Type:                  Ready
Events:

Any idea?

Thank you in advance.

user_15537201 · Answer

ApiVersions

First I noticed you're using v1alpha2 apiVersion which is depricated and will be removed in 1.6 cert-manager:

$ kubectl apply -f cluster-alpha.yaml
Warning: cert-manager.io/v1alpha2 ClusterIssuer is deprecated in v1.4+, unavailable in v1.6+; use cert-manager.io/v1 ClusterIssuer

I used apiVersion: cert-manager.io/v1 in reproduction.

Same for v1beta1 ingress, consider updating it to networking.k8s.io/v1.

What happens

I started reproducing your setup step by step.

I applied clusterissuer.yaml:

$ kubectl apply -f clusterissuer.yaml
clusterissuer.cert-manager.io/selfsigned created

$ kubectl get clusterissuer
NAME         READY   AGE
selfsigned   True    11s

Pay attention that READY is set to True.

Next I applied certificate.yaml:

$ kubectl apply -f cert.yaml
certificate.cert-manager.io/selfsigned-certificate created

$ kubectl get cert
NAME                     READY   SECRET                             AGE
selfsigned-certificate   True    hello-deployment-tls-ca-key-pair   7s

Next step is to add the second ClusterIssuer which is referenced to hello-deployment-tls-ca-key-pair secret:

$ kubectl apply -f clusterissuer2.yaml
clusterissuer.cert-manager.io/hello-deployment-tls created

$ kubectl get clusterissuer
NAME                   READY   AGE
hello-deployment-tls   False   6s
selfsigned             True    3m50

ClusterIssuer hello-deployment-tls is not ready. Here's why:

$ kubectl describe clusterissuer hello-deployment-tls
...
Events:
  Type     Reason         Age                From          Message
  ----     ------         ----               ----          -------
  Warning  ErrGetKeyPair  10s (x5 over 75s)  cert-manager  Error getting keypair for CA issuer: secret "hello-deployment-tls-ca-key-pair" not found
  Warning  ErrInitIssuer  10s (x5 over 75s)  cert-manager  Error initializing issuer: secret "hello-deployment-tls-ca-key-pair" not found

This is expected behaviour since:

When referencing a Secret resource in ClusterIssuer resources (eg apiKeySecretRef) the Secret needs to be in the same namespace as the cert-manager controller pod. You can optionally override this by using the --cluster-resource-namespace argument to the controller.

Reference

Answer - how to move forward

I edited the cert-manager deployment so it will look for secrets in default namespace (this is not ideal, I'd use issuer instead in default namespace):

$ kubectl edit deploy cert-manager -n cert-manager

spec:
  containers:
  - args:
    - --v=2
    - --cluster-resource-namespace=default

It takes about a minute for cert-manager to start. Redeployed clusterissuer2.yaml:

$ kubectl delete -f clusterissuer2.yaml
clusterissuer.cert-manager.io "hello-deployment-tls" deleted

$ kubectl apply -f clusterissuer2.yaml
clusterissuer.cert-manager.io/hello-deployment-tls created

$ kubectl get clusterissuer
NAME                   READY   AGE
hello-deployment-tls   True    3s
selfsigned             True    5m42s

Both are READY. Moving forward with certificate2.yaml:

$ kubectl apply -f cert2.yaml
certificate.cert-manager.io/selfsigned-certificate2 created

$ kubectl get cert
NAME                      READY   SECRET                              AGE
selfsigned-certificate    True    hello-deployment-tls-ca-key-pair    33s
selfsigned-certificate2   True    hello-deployment-tls-ca-key-pair2   6s

$ kubectl get certificaterequest
NAME                            APPROVED   DENIED   READY   ISSUER                 REQUESTOR                                         AGE
selfsigned-certificate-jj98f    True                True    selfsigned             system:serviceaccount:cert-manager:cert-manager   52s
selfsigned-certificate2-jwq5c   True                True    hello-deployment-tls   system:serviceaccount:cert-manager:cert-manager   25s

Ingress

When host is not added to ingress, it doesn't create any certificates and seems to used some fake one from ingress which is issued by CN = Kubernetes Ingress Controller Fake Certificate.

Events from ingress:

Events:
  Type     Reason     Age   From                      Message
  ----     ------     ----  ----                      -------
  Warning  BadConfig  5s    cert-manager              TLS entry 0 is invalid: secret "example-cert" for ingress TLS has no hosts specified

When I added DNS to ingress:

Events:
  Type     Reason             Age                From                      Message
  ----     ------             ----               ----                      -------
Normal   CreateCertificate  4s                 cert-manager              Successfully created Certificate "example-cert"

Answer, part 2 (about ingress, certificates and issuer)

You don't need to create a certificate if you're referencing to issuer in ingress rule. Ingress will issue certificate for you when all details are presented, such as:

annotation cert-manager.io/cluster-issuer: "hello-deployment-tls"
spec.tls part with host within
spec.rules.host

OR

if you want to create certificate manually and ask ingress to use it, then:

remove annotation cert-manager.io/cluster-issuer: "hello-deployment-tls"
create certificate manually
refer to it in ingress rule.

You can check certificate details in browser and find that it no longer has issuer as CN = Kubernetes Ingress Controller Fake Certificate, in my case it's empty.

Note - cert-manager v1.4

Initially I used a bit outdated cert-manager v1.4 and got this issue which has gone after updating to 1.4.1.

It looks like:

$ kubectl describe certificaterequest selfsigned-certificate2-45k2c

Events:
  Type     Reason           Age   From          Message
  ----     ------           ----  ----          -------
  Normal   cert-manager.io  41s   cert-manager  Certificate request has been approved by cert-manager.io
  Warning  DecodeError      41s   cert-manager  Failed to decode returned certificate: error decoding certificate PEM block