Observability with Prometheus and Grafana

Exploring and Interpreting Prometheus Metrics & Queries

When software is instrumented with Prometheus, it exposes metrics that can be scraped over HTTP, usually on the /metrics endpoint. Let's take two examples that Prometheus itself exposes: promhttp_metric_handler_requests_total and process_cpu_seconds_total. How can we interpret these metrics? We have two ways to understand them, either by finding them in the official documentation or viewing their description directly by querying the /metrics endpoint.

Example:

# Run this command to see the metrics exposed by Prometheus
curl -s http://localhost:9090/metrics

# Sample output:
...
...
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 881
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
...
...
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 13.16
...
...

In the above example, we can understand the metrics by looking at two fields: TYPE and HELP.

• HELP: This provides a brief description of what the metric represents. For example, promhttp_metric_handler_requests_total is described as "Total number of scrapes by HTTP status code," indicating that it counts the total number of scrape requests made to the Prometheus server, categorized by their HTTP status codes.

• TYPE: This indicates the type of the metric. In this case, both metrics are of type counter. The type is typically one of the following: counter, gauge, histogram