Observability with Prometheus and Grafana

Exporters and Metrics in Kubernetes

In a Kubernetes cluster, various components expose metrics that can be scraped by Prometheus for monitoring and alerting purposes. Some of the metric types available include:

• Node Metrics: The Kubelet, via the integrated cAdvisor, exposes container-level resource usage metrics like CPU, memory, filesystem, and network. For deeper OS-level metrics (e.g., system load, hardware info), the Node Exporter should be deployed as a DaemonSet.

• Pod Metrics: These show resource usage (CPU, memory, network) per pod and container. They are collected from the Kubelet and surfaced by tools like metrics-server for HPA and by Prometheus for alerting.

• Control Plane Metrics: Kubernetes core components such as the API server, scheduler, and controller manager expose detailed Prometheus metrics. These include request latencies, reconciliation loops, scheduling decisions, and internal error rates - essential for cluster health diagnostics.

• etcd Metrics: The key-value store behind Kubernetes exposes metrics on request latency, compaction, and leader elections, which are critical for API reliability.

• System Component Metrics: Core add-ons like CoreDNS, kube-proxy, and CNI plugins also expose metrics. These cover DNS resolution time, proxy connection stats, and network policy behavior. Some components need explicit flags to enable metric endpoints.

• Kubernetes Resource State Metrics: The kube-state-metrics service exports Prometheus metrics that reflect the desired and current state of Kubernetes objects like Deployments, StatefulSets, and Services. These aren't performance metrics but metadata, useful for alerts like "replicas mismatch".

• Autoscaler Metrics: Both the Horizontal Pod Autoscaler (HPA) and Cluster Autoscaler emit metrics describing scaling decisions, evaluation intervals, and trigger conditions.

• Custom Application Metrics: Applications can expose their own Prometheus-compatible metrics (e.g., request latency, error rates, business KPIs) via /metrics endpoints. These are essential for SLOs and root cause analysis.

• And more: PersistentVolume metrics (via CSI drivers), Ingress and service-level metrics (via Ingress controllers or service mesh), Security metrics (from tools like OPA/Gatekeeper, Kyverno, or NeuVector) ..

Most of the components expose Prometheus-format metrics on a /metrics HTTP endpoint. However, some components and add-ons do not expose by default and require enabling them with specific flags (for example, --metrics-bind-address or --enable-metrics).

You can access these metrics directly over the network (if the endpoint is exposed) or by using kubectl proxy to route API requests through the control plane securely. This is useful for inspecting metrics from components like the API server, scheduler, or controller manager without exposing their ports externally.

Run the following command on the control node (server1 in our case) to start the kubectl proxy:

kubectl proxy --port=8001 > /dev/null 2>&1 &

Then you can access the metrics at the following URL:

# We need the node name of the control node
# We have only one node in this example
export node_name=$(kubectl get nodes -o jsonpath="{.items[0].metadata.name}")
# If you have multiple nodes, replace the above command with:
# export node_name="your_node_name"

export metrics_api_url="http://127.0.0.1:8001/api/v1/nodes/$node_name/proxy/metrics"

# "/proxy/metrics"
curl $metrics_api_url

# "/proxy/metrics/cadvisor"
curl $metrics_api_url/cadvisor

# "/proxy/metrics/resource"
curl $metrics_api_url/resource

# "/proxy/metrics/probes"
curl $metrics_api_url/probes

We can also deploy the Node Exporter to the cluster to collect more metrics about the nodes. The following commands are also run on server1 (the control node):

# Install Helm
# Choose the desired version
export version="v3.19.0"

# Download the installation script
curl -fsSL -o \
  get_helm.sh \
  https://raw.githubusercontent.com/helm/helm/$version