Observability with Prometheus and Grafana

Use the Right Relabeling Option - A Cheat Sheet

Misunderstanding Prometheus relabeling stages leads to incorrect configs and wasted effort. Pick the option that matches where in the pipeline you want the change to happen.

relabel_configs (Target relabeling)

When: Before scraping, during service discovery.

Use for:

• Keep or drop targets discovered via SD.
• Rewrite target labels (e.g., __address__, job, instance).
• Derive stable target labels from discovery metadata.

Example: Drop test pods

scrape_configs:
  - job_name: k8s-pods
    kubernetes_sd_configs: [{ role: pod }]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_env]
        regex: test
        action: drop

metric_relabel_configs (Metric relabeling)

When: After scraping, before ingesting into TSDB.

Use for:

• Drop expensive or noisy series.
• Normalize or rename metric labels.
• Remove sensitive labels before storage.

Example: Drop high-cardinality paths

scrape_configs:
  - job_name: demo-app
    static_configs: [{ targets: ["localhost:8000"] }]
    metric_relabel_configs:
      - source_labels: [path]
        regex: "/api/v1/.*"
        target_label: path
        replacement: "/api/v1/*"
        action: replace

write_relabel_configs (Remote write relabeling)

When: