Using cert-manager to manage K8s certificates (Part 2)

This is part 2 of our blog post series about the cert-manager operator for RedHat OpenShift which is RedHat's delivery of the upstream Cert Manager project. Please see Using cert-manager to manage K8s certificates (part 1/2) for an introduction to the subject.

While we've used Cert Manager in part 1 of this blog post series to manage a private signing CA and how to issue signed certificates trusted only within our organization, this time we'll use Cert Manager to issue certificates signed by a trusted, third-party CA using the ACME protocol: Let's Encrypt.
In a next step we'll show how such a well-trusted certificate can automatically be provisioned as the wildcard ingress certificate for our Openshift cluster, thus getting rid of the annoying browser security pop-ups when connecting to your Openshift services.

Scenario 2: Using the Let's Encrypt issuer via ACME protocol

Using a self signing CA as described in part 1 of this blog post series might be well-suited for a limited audience of users accessing the cluster - as the CA certificate needs to be installed as a trusted CA on every end-user system, which often can be seen in managed, corporate environments.
However, if the cluster is supposed to be accessed by the public that is a no-go - so we need to use certificates issued by trusted signers. We'll explain the steps required to request and issue certificates signed by Let's Encrypt on demand using the ACME protocol and finally replace the default certificate of the OpenShift Router with a trusted one.
Just like with the Scenario 1: Using a self-signed Cluster CA in Using cert-manager to manage K8s certificates (part 1/2) we first need to create a ClusterIssuer manifest for Let's Encrypt to sign certificates.

Using up cert-manager to sign certificates using the ACME protocol requires performing the following steps:

1. Create a ClusterIssuer for Let's Encrypt
2. Use the ClusterIssuer for a Route or a regular Secret
3. Use the ClusterIssuer to create a certificate for the OpenShift Router

1. Create a ClusterIssuer for Let's Encrypt

When issuing certificates using the ACME protocol we first need to make a decision for the identifier validation. Identifier validation is basically the mechanism how the certificate issuing authority is able to verify in an automated way that the certificate requester is in control of the domain/host and holds the private key used for requesting certificates.

Identifier validation relies on challenges; the most widely used are:

• HTTP Challenge - the requestor proves his control over the domain/host by verifying that he is able to provision arbitrary HTTP resources on a web-server reachable under that domain URL. In our case the Openshift cluster must both be accessible via the internet and able to serve HTTP requests.

• DNS Challenge - the requestor proves his control over the domain/host by verifying that it can provision specific TXT DNS records for the domain. Note that this challenge type does not require public HTTP accessibility of the cluster.

  Please see the links above for more details on the challenge types.

In our case we have decided to use the DNS Challenge provider with cert-manager's CloudFlare provider, as DNS of our cluster-domain is managed via CloudFlare. But lots of other, well-known DNS providers are supported by cert-manager.

Setting up the DNS challenge using the CloudFlare provider

The CloudFlare provider offers two types of Tokens to authenticate requests, namely the API Tokens which allow application-scoped keys bound to specific zones and permissions, and API Keys which are globally-scoped keys that carry the same permissions as your account. As the API Tokens allow for more fine grained security configuration they are the recommended token type to be used.

The API Token needs to be stored in an OpenShift/Kubernetes Secret like the following:

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: <your-token> 

Note: As a best practice you should not store tokens (or any other sensitive data) in source-code management systems like Git in an unencrypted way. You should use any kind of vault implementation to manage the secret or at least use SealedSecrets to manage the Secret via Git.

Setting up the initial ClusterIssuer

The configuration of the ClusterIssuer defines the type of identifier validation (DNS challenge type), specifics of the challenge provider with an authentication secret (CloudFlare API token), the secret to store the automatically generated ACME account private key (*letsencrypt-private-key) as well as the ACME server URL.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns
  namespace: infra
spec:
  acme:
    solvers:
      - dns01:
          cloudflare:
            email: me@my.email.domain
            apiTokenSecretRef:
              name: cloudflare-api-token-secret
              key: api-token
          cnameStrategy: Follow
    privateKeySecretRef:
      name: letsencrypt-private-key
    server: https://acme-v02.api.letsencrypt.org/directory

Note 1:
The ClusterIssuer.spec.acme.server URL needs to be taken from the ACME providers documentation. For Let's Encrypt the Let's Encrypt ACME Directory URL is https://acme-v02.api.letsencrypt.org/directory.
Note 2:
CloudFlare provider offers two API token types which are referenced differently in the ClusterIssuer.spec.acme.solvers.dns01.cloudflare section of the manifest. Using the wrong field results in errors like the following (see: #2036) :

Error:
  6003: Invalid request headers\u003c-
  6103: Invalid format for X-Auth-Key header" "key"="xxxxxx"

Apply the above manifest and wait for the operator to both create the ClusterIssuer letsencrypt-dns and set its status to Status: True and Type: Ready.
Make sure to run:
oc -n infra describe clusterissuers letsencrypt-dns
to verify its status before proceeding!

2. Use the new ClusterIssuer for Routes or regular Secrets

This ClusterIssuer can now be used to obtain trusted, signed certificates from Let's Encrypt. These certificates can secure Openshift Routes or be used in regular K8s Secrets to secure applications or an Ingress via TLS encryption.

Using an Ingress/Route certificate

Openshift provides the Route resource which lets you access applications running in the cluster from the outside - just like K8s Ingress. A Route however does not allow to refer to an existing K8s secret, which might change over time (e.g. when cert-manager does a certificate renewal). To be able to use cert-manager-issued certificates to encrypt the traffic between the client and the application running on OpenShift we need to create an Ingress object. Openshift by default watches Ingress objects continuously and automatically creates or updates the dependent Route object for you.

Long story short - we just need to add a few annotations to the Ingress object which allows us to trigger the certificate creation process, resulting in a HTTPS-secured Route.

To provide a TLS Route to an application (here we use the very simple hello-openshift application) we have to create a manifest for an Ingress which we annotate with the ClusterIssuer information (L7-9) as shown below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-openshift-letsencrypt
  namespace: infra
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-dns # <==
    cert-manager.io/duration: 8760h                 # <==
    cert-manager.io/renew-before: 360h              # <==
    haproxy.router.openshift.io/timeout: 15m  # custom annot. will be copied to the Route!
spec:
  ingressClassName: openshift-default
  tls:
    - hosts:
      - le-hello-openshift.apps.lab.cloud.example.com
      secretName: hello-le-crt   # <== created by cert-manager!
  rules:
    - host: le-hello-openshift.apps.lab.cloud.example.com
      http:
        paths:
        - backend:
            service:
              name: hello-openshift
              port:
                number: 8080
          path: /
          pathType: Prefix

Once applied, this Ingress specification will:

• trigger the creation of a certificate secret named hello-le-crt for host provided via Ingress.spec.tls.hosts. This certificate will be signed by the ClusterIssuer linked via the cert-manager.io/cluster-issuer annotation. Note that we are using additional, optional cert-manager annotations:
  • cert-manager.io/duration to specify the validity of 1 year (365 * 24 hours)
  • cert-manager.io/renew-before to have our certificate be renewed 15 days (15 * 24 hours) before expiration.
• create a Route which uses the hostname from Ingress.spec.tls.hosts and the certificate's public and private key-pair obtained from the hello-le-crt secret.

Using cert-manager certificates in your application

If your application takes care of encrypting traffic itself (ie. using a secure pass-though route), you can get a TLS certificate (stored in a secret, of course) by applying a Certificate resource like the following:

kind: Certificate
apiVersion: cert-manager.io/v1
metadata:
  name: hello-le-crt-app
  namespace: infra
spec:
  commonName: hello-openshift.apps.lab.cloud.example.com  # optional when SAN (dnsNames) are used!
  # isCA: false
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt-dns
  privateKey:
    algorithm: ECDSA
    size: 256
  secretName: hello-le-crt-app
  duration: 8760h0m0s
  renewBefore: 360h0m0s
  dnsNames:
    - "hello-openshift.apps.lab.cloud.example.com"
    # add other DNS names to this list

Once the manifest is applied, we'll get the secret hello-le-crt-app created, containing a tls.key and tls.crt data field. We can then mount it into our application pod(s) as usual, or use them in any other way we can use certificates in OpenShift/Kubernetes.

Note: If the certificate is volume-mounted to a pod and gets renewed while the pod is running, the pod usually needs to be restarted to pick up the renewed certificate!

3. Use the ClusterIssuer to create a certificate for the OpenShift Router

To replace the Router' default self-signed certificate with a certificate signed by a trusted CA (Let's Encrypt) the following steps are required:

1. Create the Let's Encrypt signed certificate for the OpenShift Router
2. Update to default IngressController in the openshift-ingress-operator namespace

Create the Let's Encrypt signed certificate for the OpenShift Router

The first step to replace the Router certificate with a certificate signed via the ACME protocol is to create a Certificate in the openshift-ingress namespace, putting your wildcard cluster app-domain name *.apps.lab.cloud.example.com into both spec.commonName and spec.dnsNames attributes.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: le-ingress-certificate
  namespace: openshift-ingress
spec:
  commonName: "*.apps.lab.cloud.example.com"
  dnsNames:
    - "*.apps.lab.cloud.example.com"
    # add other DNS names to this list
  renewBefore: 240h0m0s
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt-dns
  privateKey:
    algorithm: ECDSA
    size: 256
  secretName: le-ingress-certificate
  subject:
    countries:
      - AT
    localities:
      - Vienna
    organizations:
      - alpium-it
    postalCodes:
      - '1090'
    provinces:
      - Vienna
    streetAddresses:
      - "Liechtensteinstrasse 111-115"

Update to default IngressController in the openshift-ingress-operator namespace

To have the Openshift router actually use the newly created TLS certificate, we need to update the default IngressController manifest in the openshift-ingress-operator namespace with the Secret created from the Certificate named le-ingress-certificate in the openshift-ingress namespace. To do this, simply apply the following manifest:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: default
  namespace: openshift-ingress-operator
spec:
  defaultCertificate:
    name: le-ingress-certificate

Note: You will need to wait a few moments while the IngressController operator triggers a rolling restart of the router pods running inside the openshift-ingress namespace.

Summary

In this blog post we have shown how to use OpenShift cert-manager to issue certificates signed by a trusted signer using the ACME protocol via a DNS challenge, and how to use certificates for Routes to secure applications on Openshift. Furthermore we have demonstrated how to set up a publicly trusted certificate for the OpenShift Router's wildcard apps-domain.

zurück