Securing and Managing Secrets in Kubernetes: 3 Key Practices

Securing and managing secrets in Kubernetes is a crucial aspect of maintaining the security and integrity of your clusters and apps. Kubernetes secrets are used to store sensitive information such as passwords, tokens, and keys, and are often used to authenticate and authorize access to various resources within a cluster.

However, as a default, Kubernetes secrets are only encoded in base64, which is not a secure way to handle the secrets in a production or an enterprise application. The threat of them being exposed still remains top of mind.

In order to solve this problem, we need to implement proper security measures to protect secrets in Kubernetes clusters. The prime goal of using secrets is to store information we do not want exposed, information as:

• Database Credentials: It can be a database url, database password and username. 
• Tokens: API Keys, Firebase tokens, etc.
• Keys: AWS Secrets keys, Twilio Keys, SES SMTP details, etc. 

Kubernetes secretes are stored in plaintext in etcd, the distributed key-value store that is used by Kubernetes to store configuration data. While etcd can be encrypted, access to etcd should still be restricted by using secure storage backends and restricting access to only the users and applications that need them.

Here are three key practices for securing and managing secrets in Kubernetes using different mechanisms and tools to achieve higher level of cluster security.

Use a key management system

KMS tools can help protect secrets in Kubernetes by providing a secure storage backend for secrets, and controlling the access to those secrets.

A KMS can be used in conjunction with Kubernetes to provide an additional layer of security for sensitive information used by the applications running in a Kubernetes cluster. By storing secrets in a KMS, rather than in plaintext in a Kubernetes cluster, secrets are encrypted and protected by the KMS, and access to the secrets can be controlled via the KMS.

When we are creating multiple yaml files for Kubernetes deployments or helm-charts with multiple dev, prod and staging environments, we have to define our secrets in value files. And these files are eventually stored in a repository. 

So to avoid this, third-party KMS tools help a lot as they provide more features apart from encryption, like control access to secrets, role-based access control, token-based authentication, and automatic rotation of secrets.

These kind of features help make our production infrastructure more secure, less prone to attacks, while maintaining a zero-trust policy and also complied by PCI DSS benchmarks.

Let us take an example to understand how these features help in a real use-case; A developer works in an organization, having access to the cluster. When he leaves the job the company needs to manually change all creds, but with features like temporary creds and automatic rotation you can be risk free.

Implement REST-ENCRYPTION

REST-based encryption is a technique for encrypting sensitive data in a RESTful API (Representational State Transfer), which can be implemented in Kubernetes to establish a secure communication between APIs.

The Kubernetes API server is the central component of a Kubernetes cluster, responsible for managing the cluster’s resources, such as pods, services and an identity provider — all used to protect secret data in etcd, which provides no encryption because it is stored as plain text in a key-value format. 

Hence, by creating an EncryptionConfiguration you can make your configmaps and secrets encrypted in an k8s cluster. 

EncryptionConfiguration is a configuration object in Kubernetes that allows for the management of encryption-at-rest for etcd, the datastore used by Kubernetes to store its configuration data. 

It allows you to specify the encryption providers, key providers, and encryption context that should be used to encrypt the data stored in etcd. This feature can be used to ensure that the data stored in etcd is protected and secure, even in the event that the etcd cluster is compromised

(Ref: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/)

Audit and monitor

Implementing, auditing and monitoring secrets in Kubernetes is important for ensuring the security and integrity of secrets in a Kubernetes cluster.

There are several ways that auditing and monitoring of secrets can be implemented in Kubernetes:

Auditing API requests

The Kubernetes API server logs all requests made to it, including requests to create, update, or delete secrets. By enabling auditing in the API server, organizations can track changes made to secrets and identify any suspicious activity.

Using a secrets management tool 

Many secrets management tools, such as Hashicorp Vault and Akeyless (the former open-source, the latter SaaS), have built-in auditing and monitoring capabilities. Both of these tools provide features like secure backends, dynamic creds, rotation, access-control, etc. — and have their own pros and cons:

Hashicorp Vault: Widely used, open-source, very well documented, integration of consul service mesh. But its implementation is also very complicated as it involves a lot of time to study the tool. 

For instance, in Kubernetes when trying to implement Vault via its webhook feature to inject secrets inside the Pod, you will not find a concrete and easy solution while following the official documentation. You have to look for a work-around. For me Banzai Cloud (third-party) vault-operators helped to implement it. 

Also trying to debug any issue with a self-hosting vault proves to be extremely hard to find logs of init containers and agents. 

On the other hand,

Akeyless: Less known than Hashicorp Vault, a paid alternative that provides most of the Vault features. Since it’s a SaaS, there’s no need to worry about setup and maintenance.

Implementing role-based access control (RBAC)

RBAC allows organizations to define fine-grain permissions for users and groups, and can be used to limit who has access to secrets in a Kubernetes cluster. By implementing RBAC, organizations can ensure that only authorized users have access to secrets, and can track access to secrets through the API server logs.

Conclusion

Managing secrets in a Kubernetes cluster is an important aspect of ensuring the security and integrity of the cluster, especially with production environments. By adhering to the above mentioned key practices, one can surely make their environment safe and secure.