Troubleshooting Kubernetes Networking — Part1

Basics

Services

Before we proceed, let’s check about Kubernetes service, A Kubernetes service is a logical abstraction for a deployed group of pods in a cluster (which all perform the same function). Since pods are ephemeral, a service enables a group of pods, which provide specific functions (web services, image processing, etc.) to be assigned a name and unique IP address (clusterIP). As long as the service is running that IP address, it will not change. Services also define policies for their access.

Types of Kubernetes services?

• ClusterIP. Exposes a service which is only accessible from within the cluster.
• NodePort. Exposes a service via a static port on each node’s IP.
• LoadBalancer. Exposes the service via the cloud provider’s load balancer.
• ExternalName. Maps a service to a predefined externalName field by returning a value for the CNAME record.

Container Network Interface

As we know, Every Pod in a cluster gets its own unique cluster-wide IP address. This means you do not need to explicitly create links between Pods and you almost never need to deal with mapping container ports to host ports. This creates a clean, backwards-compatible model where Pods can be treated much like VMs or physical hosts from the perspectives of port allocation, naming, service discovery, load balancing, application configuration, and migration.

Kubernetes imposes the following fundamental requirements on any networking implementation (barring any intentional network segmentation policies): pods can communicate with all other pods on any other node without NAT agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that node.

Kubernetes networking addresses four concerns:

• Containers within a Pod use networking to communicate via loopback.
• Cluster networking provides communication between different Pods.
• The Service resource lets you expose an application running in Pods to be reachable from outside your cluster.
• You can also use Services to publish services only for consumption inside your cluster.

CNI Plugins:

Here are some familiar or most used CNI plugins,

1. Flannel
2. Calico
3. Cilium
4. WeaveNet
5. Canal

Summary Matrix

How to choose a CNI Provider?

There is no single CNI provider that meets all our project needs, here some details about each provider. For easy setup and configuration, Flannel and Weavenet provide great capabilities. Calico is better for performance since it uses an underlay network through BGP. Cilium utilizes a completely different application-layer filtering model through BPF and is more geared towards enterprise security.

Basic Troubleshooting

Traffic

As we have seen, Kubernetes supports a variety of networking plugins as seen above and each fails in its own way. To troubleshoot the issues in networking we should understand the core. Kubernetes relies on the Netfilter kernel module to set up low level cluster IP load balancing. This requires two critical modules, IP forwarding and bridging.

Kernel IP forwarding

IP forwarding is a kernel setting that allows forwarding of the traffic coming from one interface to be routed to another interface. This setting is necessary for Linux kernel to route traffic from containers to the outside world.

What it causes?

Sometimes this setting could be reset by a security team running while security scans/enforcements or some system changes, or have not been configured to survive a reboot. When this happens networking starts failing.

Pod to service connection times out:

Tcpdump could show that lots of repeated SYN packets are sent, but no ACK is received.

How to diagnose

Check that ipv4 forwarding is enabled

Continue reading it on https://foxutech.com/troubleshooting-kubernetes-networking-part1/

You can follow us on social media, to get some short knowledges regularly.

• Facebook: https://www.facebook.com/foxutech/
• Instagram: https://www.instagram.com/foxutech/
• YouTube: https://www.youtube.com/c/FoxuTech/
• Twitter: https://twitter.com/foxutech
• Medium: FoxuTech — Medium