Securing Kubernetes: The Paranoid Guide

She says that one of the reasons why you might want to do penetration testing is stories such as this.

In 2018, Tesla left their Kubernetes Dashboard open to the internet. The Dashboard has cluster-admin privileges.

They were hacked, and the end result was their system was used to run cryptocurrency mining malware.

It was a big headline and one that prompted the larger Kubernetes industry to focus more on security.

But why?

How did one of the biggest tech companies in Silicon Valley got hacked?

Is it simply a human issue? Or is there more to Security in Kubernetes?

I'm your host Kassandra Russel, and today we are going to talk about Security in Kubernetes.

We will examine the differences between securing a traditional environment and a container-based environment.

Next, we will discuss industry standards and emerging thought patterns around security.

And finally, we will go through some of the best security practices and general security advice for production workloads in Kubernetes.

Before diving into all of this, we’ve been busy during the last weeks working on a new project. If you like this podcast, you will certainly like the new project, it’s a surprise, we are going to talk more about it in the future. In the meantime, you can subscribe to the podcast announcement list, we will announce it soon.

Back to the subject at hand, remember the two generals' problem from one of our previous episodes?

It's a classic thought experiment exposing an unsolvable problem and demonstrating the design challenges of distributed systems and the pitfall of reaching consensus over a lossy network.

If you are interested in knowing more about this, we recommend you listen to our 5th episode “The Ubiquity of Kubernetes”.

Now, imagine the two generals somehow were able to get their communications across and agreed to attack the enemy castle at the same time.

This time as they are both nearing the castle, they saw layers and layers of defensive structures.

Wave after wave, the soldiers under the command of the two generals are trying to enter the castle. However, since there are multiple defensive structures, they were trapped in the middle of these layers and were unable to conquer the castle.

Why is this relevant to security? According to Ian Lewis, a developer advocate at Google Cloud. Security is a spectrum. There are many different layers to improve security on. For example, you can improve the security of your runtime, security of your host, and security of your network layer.

This is similar to how multiple layers of defensive fortification allowed the castle defenders to win and repel the invaders ultimately. Multiple security layers enable systems to be resilient and provide as many hurdles between the attacker and the targeted asset.

Thinking about it this way allows us to expound on more security concepts.

For example, how big is the castle? How many attackers can go in at once?

In the security world, we call this attack surface or attack vector.

To explain it in a sentence, the attack surface is where your hackers can attack your systems. It is measured by how big your attack surface is.

One place the hackers can attack is the operating system; the more the packages and libraries you have, the bigger your attack surface is.

Going back to the castle, imagine if the two generals didn't come to a conclusion and therefore did not attack the castle. The castle would be safe automatically.

In the security world, understanding who attacks your castle is determined by your threat model.

There are many kinds of threat models; let's discover three of them.

The first threat could be a random hacker on the internet. But how? If I didn't disclose the IP or DNS of my service, they could not know it, right?

Well, not exactly.

There are multiple scanners and crawlers that look at IP addresses and DNS names and make them publicly available via a search engine. This means putting your system on the internet will get it scanned at some point.

Security by obscurity is an illusion.

The second type of threat could be a targeted attacker. This is especially prevalent in cases where your system contains valuable information that could lead to monetary incentives.

For example, if you are running banks that contain financial information, then you could be facing targeted attackers. They usually run more sophisticated attacks and spend more time to break into your systems.

This is opposed to random hackers who usually have an automated script, and if they fail, they move on.

Targeted attackers are also likely to be manually doing discovery of your systems, so it's especially harder to defend against them.

Finally, there are nation-states hackers. They usually work for governments and have a license to Hack. Motivated by nationalism, their mission is to disrupt other nations and their organizations via cyberspace. If a nation-state targets you, then you are most likely to have serious problems.

Before jumping to the next part, we'd like to invite you to subscribe to The Chief I/O newsletter.

By subscribing, you'll get your weekly dose of cloud-native insights and trends in your inbox twice a week.

Monday's email will include the latest trends and news, while Tuesday's email will include the best stories and tutorials from our publication.

Visit "thechief.io/newsletter" and join thousands of other subscribers today!

According to a survey commissioned by StackRox, nearly 94 percent of the respondents have experienced security incidents in their container environments in the past 12 months.

Securing container-based environments is becoming increasingly important, which includes Kubernetes.

So how would these traditional security principles apply to Kubernetes?

Let's first talk about the Kubernetes attack surface.

Where in Kubernetes is an attack most probable?

Usually, the first point of attack would be either the Kubernetes API Server, Kubelet, or the ETCD.

Just before version 1.10, the API Server has an unauthenticated port.

API Server, as discussed in one previous episode, is the brain of your cluster. It is the coordination system upon which all other components use to coordinate.

ETCD contains all the Kubernetes states that include the secrets, and by default, those secrets are not encrypted.

Kubelet is the daemon that runs inside the server. This means that it has direct access to the underlying host.

If any hacker could break into either of these three components, it's safe to say:

When we think about securing Kubernetes, securing these three components is not enough; you have to think holistically about all the layers that make up the entire system.

For example, you also have to think about the host OS, which OS does your Kubelet run?

Another could be networking. Are you hosting the cluster in a publicly accessible network?

Containers' runtime could also be compromised; which container OS is your applications running and which packages and libraries your application relies on?

Adding an additional overhead of containers into an already complex security model is challenging. This is why a lot of companies rely on security standards, such as the NIST cybersecurity framework, as well as CIS benchmarks for Kubernetes.

Speaking of containers you need to make sure that the supply chain of your containers is secure.

Let's try another thought experiment.

How are pizzas made? How do they maintain the freshness of the ingredients when you order one?

The dough, meats, cheese, and most importantly, the pineapple would probably all arrive at the same time fresh.

They would have their own network of vendors and suppliers that we could trace all the way back to the land and farming.

What happens if one of those suppliers, all of a sudden, got compromised? Let's say, for example, pineapple yields started falling!

The end result is that the pizza would probably not taste as good without the pineapple.

This is the same as containers. You wouldn't want any part of the container supply chain compromised.

Containers are harder to secure because they are usually built on top of another container, which was probably built on top of another container, and so on.

Understanding your container's entire build is hard. This is why most companies have a set of preapproved base containers. Similarly, there is usually a preapproved set of ingredients ready to go in your local domino's store. Meaning the chain of containers is already approved and ready to go.

Following this should narrow your attack vector in the supply chain. This is one of the industry-standard laid out in the Center for Internet Security(CIS) industry benchmarks.

Speaking of the CIS, benchmarks answer the following question: How secure is your cluster?

There are also useful resources around to remediate a specific problem that the benchmark identifies.

Here at FAUN, we recommend you start by using CIS industry benchmarks. There are three, in particular, you have to pay extra attention to.

First is the OS CIS benchmarks, this depends on which Operating System your Kubernetes host is running on top of. A good example of this is CIS Benchmark for Debian OS for all the folks running Debian Kops Kubernetes installation.

The second is the Docker Benchmarks; on this one, you should probably focus on image creation standards. The trick is to find which parts identify with your system.

Lastly and most importantly is the Kubernetes CIS benchmarks. This benchmark contains a myriad of information around how to secure your cluster but, at the same time, why you should add those controls.

Once you have the CIS Benchmark results for your cluster, the next thing is to identify mitigation or remediation.

Another question your security team might be interested in is how mature your security controls and operations in a container world are?

Usually, you start by going through the high-level cybersecurity framework NIST provides, and then you dig deeper into the relevant parts of the framework.

NIST, the National Institute of Standards and Technology, provides a framework for Detection, Identification, Protection, Response, and Recovery operations.

The primary aim is to use the controls you have put in place and start using the framework to manage risk.

Using NIST as your jump-off point allows you to identify best practices and industry standards. For example, a special publication NIST-800-190, known as Application Container Security Guide, gives you a good overview of how to secure containers from a NIST point of view.

As more workloads get migrated to Kubernetes, it is increasingly important that we take steps to ensure the security of Kubernetes.

Today we have discussed some key topics in this area. If you would like to hear more about this topic or any other topic, please email us at community@faun.dev.

Don't forget to follow @joinFAUN on Twitter and rate this podcast on the platform you’re using. Thank you for listening and take care!