Security Challenges of Developing Kubernetes Infrastructure with IaC

Yet many developers may find themselves hating Kubernetes because it is, in a word, hard. The tradeoff for Kubernetes’s enormous power and flexibility is its complexity. Mastering that complexity is no mean feat.

Not only that, but navigating the security challenges posed by Kubernetes’s complexity is also enormously difficult. Small configuration mistakes or oversights within Kubernetes can have profoundly negative security consequences.

To illustrate the point, let’s walk you through a brief Kubernetes security overview, which addresses six common security mistakes in Kubernetes that are all too easy to make. We’ll focus in particular on the types of security issues that can become widespread within a cluster if you manage the cluster via infrastructure as code (IaC). This approach means that a security misconfiguration in a single IaC file can be quickly reproduced across an entire environment. Below are some of the common Kubernetes misconfigurations and security challenges when it comes to writing manifests and developing Kubernetes infrastructure.

Kubernetes containers running in privileged mode

Running containers in privileged mode—which gives them unfettered access to host-level resources—is like giving a redshirt the captain’s chair: it’s granting way too much power. And if you do, you may be unhappy with the consequences.

Unfortunately, running containers in privileged mode is a configuration mistake that you can easily make when configuring Kubernetes deployments. It just takes one variable/value: privileged:true.

Ideally, you’ll avoid this mistake by not configuring containers to run in privileged mode in the first place. But you can also audit your Kubernetes configurations using a tool like Checkov to find containers that were deployed in privileged mode.

Allowing container privilege escalation

Along similar lines, there’s really no situation in which Kubernetes containers should be allowed to escalate into privileged mode. To preclude containers from running in privileged mode, make sure to set “allowPrivilegeEscalation:” to “false” when configuring security contexts. Note that “allowPrivilegeEscalation” is automatically set to true if the container is privileged (see previous) or given CAP_SYS_ADMIN capabilities. Avoid those as well.

Failing to encrypt internal traffic

Another small but critical setting is the --kubelet-https=... flag. It should be set to “true.” If it’s not, traffic between your API server and kubelets won’t be encrypted.

Omitting the setting entirely also usually means that traffic will be encrypted by default. But because this behavior could vary depending on which Kubernetes version and distribution you use, it pays to be explicit about requiring Kubelet encryption, at least until the Kubernetes developers encrypt traffic universally by default.

Pulling containers using the “latest” tag

When specifying how to pull Kubernetes container images from a registry, avoid settings like this:

This is risky because it tells Kubernetes to use images tagged “latest.” Those are risky for two reasons. One is that, in some cases, they may be beta versions of an app. The other is that when you pull the “latest” image, it’s harder to know exactly which version you’re using—and, by extension, it’s harder to know whether existing and new vulnerabilities affect your version or if someone has included malware in a supply chain attack. Instead, explicitly define the version of the image you want to pull (like alpine:3.15.0) or, better yet, the manifest (like alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300) to ensure that the image you want is the one you get.

Using untrusted Helm charts

Speaking of risks associated with upstream software components like container images, here’s a Kubernetes security mistake that’s easy to make and, thus, easy to reproduce as part of code-driven development workflows: relying on Helm charts that may contain misconfigurations or container images with security vulnerabilities that can compromise the security.

And when we say “may contain security misconfigurations or vulnerabilities,” we might just as well say “almost certainly contain security misconfigurations or vulnerabilities.” We found that 71% of publicly available Helm charts contained at least one misconfiguration

We’re not saying you should never use third-party Helm charts. Helm charts simplify deploying and managing services in Kubernetes. When used securely, Helm presents a way to scale deployments by packaging dependencies and best-practice-based default settings. That said, you should make sure you are baking in those security best practices before you deploy them.

Forgetting to specify resource limits

You may think of resource limits in Kubernetes as a way to control infrastructure costs and prevent “noisy neighbor” issues between pods by restricting how much memory and CPU a pod can consume.

Resource limits do those things. But they are also crucial from a security perspective because they help mitigate the risk of denial-of-service threats. A compromised pod can do more damage when it can suck up the entire cluster’s resources, depriving other workloads of functioning properly.

From a security perspective, then, it’s a good idea to define resource limits in Kubernetes.

···

There’s an old saying about not sweating the small stuff. When it comes to Kubernetes security, however, it’s often the small stuff that’s really important. Small configuration errors can become gaping security holes when baked into your environment’s configuration.

Avoid these risks by enforcing Kubernetes best practices that help developers create secure configurations from the start. But because even the best developers sometimes make mistakes, implement a “belts and braces” approach by scanning your configurations to catch Kubernetes security issues that may have been overlooked.