Applying Kubernetes security best practices to Helm charts

They make scaling deployments internally and externally easier, as Kubernetes manifests and commands are bundled together with pre-vetted default values. As a user of a Helm, open source packages, known as charts, are readily at your disposal in repositories such as Artifact Hub. Launched in 2019, Artifact Hub has thousands of charts to quickly deploy pre-configured services such as databases into your cluster.

However, our recent research found that over 70% of Helm charts in Artifact Hub contained misconfigurations, such as not setting resource limits and running root containers, that violated CIS Kubernetes Benchmarks. Even the most popular dependency chart, PostgreSQL, contained multiple misconfigurations such as forgetting to add allowPrivilegeEscalation = false, which if left out defaults to true.

Part of the problem is that securing Helm charts is not as straightforward as securing a Kubernetes manifest.

What are Helm charts?

We should start with an exploration at a high level into how Helm charts work. Without Helm charts, you would write multiple Kubernetes manifests (YAML files) to take an application such as a container image, deploy it to your cluster and manage the additional settings of that image and how Kubernetes manages the container.

For example: If you want an ingress gateway, you may need to deploy the NGINX proxy with a certain number of replicas, set it as the ingress gateway and configure applications to interface with it. That alone could be three different manifests with default values across all three.

Instead, Helm bundles those manifests together, including dependencies like monitoring tools, and includes a central location for default but configurable values (values.yaml). If you come from the Terraform world, think of this as your vars.tf file.

Prior to deploying, helm install will construct the Kubernetes manifests and deploy them for you, just like kubectl apply.

How does Kubernetes security apply?

You may have already picked up on it, but by the time Kubernetes sees the Helm chart, it looks like just another manifest. That’s why most (we’ll talk about the exceptions later) of the misconfigurations for Kubernetes manifests are still relevant for Helm charts. Helm can still be set up to deploy a container as root or with PID 1.

However, it’s not as straightforward as scanning Kubernetes manifests. For example, the CIS Kubernetes Benchmarks 5.2.6 says to “minimize the admission of containers with added capabilities.” Even if there is a YAML line for allowedCapabilities, it may look like:

We’d have to dig into the values.yaml to find the right line:

Then we’d discover that the Helm chart is granting additional capabilities to perform networking tasks it maybe shouldn’t be.

The open source tool Checkov addresses this by identifying Helm charts and first converting them into Kubernetes manifests using the helm template command. The resulting manifests can be checked for misconfigurations, like any other Kubernetes YAML files.

There are exceptions. For example, the way Helm deploys pods to namespaces makes it irrelevant to check for deploying to the default namespace, which is a violation of CIS Kubernetes Benchmark 5.7.4. Also, after Helm v3, there is a Kubernetes policy to not include Tiller in any Kubernetes manifests, as it was overprivileged. However, Tiller is unlikely to show up in a Helm chart, so it is not a necessary check for Helm charts.

How to enforce Helm security in an automated way

Helm has simplified the way we deploy and manage services in Kubernetes. It presents a way to scale deployments by packaging dependencies and best-practice-based default settings. This can also scale securing those services by ensuring the defaults do not contain misconfigurations.

However, today most charts are not secure by default. This makes it important for developers to take the responsibility to secure Helm charts before deploying them into their production environments.

The best solution is to check for misconfigurations in your charts early and often. Start by checking the Helm charts you download for misconfigurations. Fix the issues that don’t meet your requirements and over time address the remaining misconfigurations. Helm simplifies this process with versioning and rollbacks if security patches break the deployment.

Keeping all the misconfiguration fixes in Helm charts enforces their scalability, so the next team that deploys the service for their use case will have secure defaults. The result is a lower attack surface for your Kubernetes applications.

To make our research dynamic and repeatable, we built and open-sourced a lightweight Helm Scanner that you can leverage to scan your own Helm charts.