Kubernetes in-cluster traffic encryption using cert-manager

Simplified recap of certain concepts

Certificates

A certificate on a server has 2 parts — the private key and the public certificate. The public certificate is sent to the client which has the identity of the server. The client “trusts” that the certificate offered to it is genuine and not a malicious masquerader by checking whether it trusts the signer/certificate authority of the certificate. Well known certificate authorities share the public certificates of their signing key to popular operating systems and browsers.

Self-signed vs CA signed certificates

If a certificate is self-signed, the only way to trust it securely is to share the public certificate with the client beforehand which has scalability challenges when multiple services are present. A CA signed certificate on the other hand only requires the public certificate of the CA to be distributed beforehand and all certificates signed using that CA are then implicitly trusted.

Why can’t we use a well known free CA like Let’s Encrypt

Well-known CA’s use the ACME protocol while generating automated certificates that contains a validation step to check whether the certificate requestor is a rightful owner of the host for which the certificate is being requested. The hosts in the use case of services running within a Kubernetes cluster are all local and cannot be validated by an external authority.

Scenario Conditions

• Host a mongodb cluster in namespace mongo with TLS encryption enabled.
• Multiple clients of the mongodb cluster running in different namespaces of the cluster.

Solution

Prerequisites

• Deploy cert-manager. For the scope of this blog, let’s assume the namespace in which it is deployed to be cert-manager
• Deploy trust in the same namespace as cert-manager
• Have an understanding of Issuers and ClusterIssuers. You can read about them here

Creating an issuer for the CA certificate

The key pair of the root CA needs to be created. To implement it in a model that is scalable across multiple environments, it is best to have cert-manager create it. Deploy the following manifest

Create a CA certificate

Now that there is an issuer, we can use it to create the CA certificate. Deploy the following manifest.

The prominent difference between this and server certificates is the isCA flag. This certificate will not be used by a server but just to sign certificates. This certificate creates a secret ca-secret in the cert-manager namespace which will be used to create all server certificates. Since this is a self-signed certificate the value of tls.crt and ca.crt will be the same. This is the file that needs to be distributed to all clients.

Distribute the CA certificate’s tls.crt to establish trust to all namespaces

Cert-manager has another tool called trust mentioned in the prerequisites above. Trust is used to distribute public keys securely across namespace and keep them in sync. In our scenario we need to share the ca.crt of the ca-secret secret in cert-manager namespace to all namespaces present and future. Also, the ca.crt is non-sensitive data so can be stored in a configmap. Deploy the following manifest

This will create a configmap called cluster-ca in each namespace with the data containing a key ca.crt with the public certificate of the root CA.

Create a ClusterIssuer for server certificates

Deploy the following manifest

This manifest creates a ClusterIssuer which can be used to generate certificates across all namespaces. It uses the key pair in the ca-secret secret.

Request server certificates and consume them in the server application

Now that the complete scaffolding is ready, actual certificates can be generated. Deploy the following manifest

This certificate will write the key pair to a secret my-cluster-tls. This certificate though in a different namespace than the actual CA key pair can use it for signing since it is exposed using a ClusterIssuer resource. There is no need to have any other self signed certificates in the cluster, hence the ca-issuer was a resource of type Issuer scoped to the cert-manager namespace.

The certificate has generated the key pair. This can now be mounted into the cluster members to serve traffic with encryption.

Access the server resources from the client by trusting the CA certificate

All clients connecting to the mongodb cluster need to mount the cluster-ca configmap in their namespace and include it as the CA certificate in the connection parameters. The exact syntax will depend upon the client technology.

This way we can have a cluster wide encryption in transit without the complexities of a service mesh. Any feedback on this article is welcome.