Observability with Prometheus and Grafana

Kubernetes Service Discovery for Prometheus

Kubernetes, the popular distributed container orchestration platform, provides a dynamic environment where resources like pods (with containers), services, and nodes can be created, modified, and destroyed frequently and dynamically. To effectively monitor such a dynamic environment, Prometheus uses the Kubernetes service discovery mechanism to automatically discover and scrape metrics from these resources.

There are 6 different roles of service discovery mechanisms available for Prometheus:

• Node: The VMs that run the Kubernetes cluster.
• Service: The services that expose the applications running in the cluster.
• Pod: The pods that run the containers in the cluster.
• Endpoints: The endpoints that are associated with the services in the cluster.
• EndpointSlice: The scalable and extensible alternative to Endpoints.
• Ingress: The Ingress resources that manage external access to the services in the cluster.

These discovery mechanisms can be configured in the Prometheus configuration file using the kubernetes_sd_configs section. For example, to scrape metrics from all pods in a specific namespace, the following configuration can be used:

scrape_configs:
  - job_name: "kubernetes-pods"

    kubernetes_sd_configs:
      - role: pod

    relabel_configs:
      # Map all Kubernetes pod labels to Prometheus labels
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)

      # Replace the 'namespace' label with the value from Kubernetes namespace metadata
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace

      # Replace the 'pod' label with the value from Kubernetes pod name metadata
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

For pods, the available meta labels are:

• __meta_kubernetes_namespace: The namespace of the pod object.
• __meta_kubernetes_pod_name: The name of the pod object.
• __meta_kubernetes_pod_ip: The pod IP of the pod object.
• __meta_kubernetes_pod_label_: Each label from the pod object, with any unsupported characters converted to an underscore.
• __meta_kubernetes_pod_labelpresent_: true for each label from the pod object, with any unsupported characters converted to an underscore.
• __meta_kubernetes_pod_annotation_: Each annotation from the pod object.
• __meta_kubernetes_pod_annotationpresent_: true for each annotation from the pod object.
• __meta_kubernetes_pod_container_init: true if the container is an InitContainer.
• __meta_kubernetes_pod_container_name: Name of the container the target address points to.
• __meta_kubernetes_pod_container_id: ID of the container the target address points to. The ID is in the form ://.
• __meta_kubernetes_pod_container_image: The image the container is using.
• __meta_kubernetes_pod_container_port_name: Name of the container port.