Observability with Prometheus and Grafana

The Graph Page

The graph page is where you can query the data and visualize it. You can access it by clicking on the Graph tab of the Query menu. Test some basic PromQL queries like the following:

up

process_cpu_seconds_total

promhttp_metric_handler_requests_total

node_cpu_seconds_total

The following table summarizes the queries above and their descriptions:

You can also run more complex queries that involve filtering, grouping, aggregating, and querying functions.

ℹ️ Filtering is the process of selecting only the data that meets certain criteria (e.g., my_metric{label="value"}).

ℹ️ Aggregating is the process of calculating a single value from multiple values, like the sum, the average, or the count (e.g., sum(my_metric)).

ℹ️ Grouping is the process of combining the data into groups based on certain criteria (e.g., sum by (group) (my_metric)).

ℹ️ Querying functions are functions that can be used to manipulate the data, like rate, increase, or predict_linear (e.g., rate(my_metric[5m])).

To better understand querying, let's start with a simple query that uses the promhttp_metric_handler_requests_total metric. This metric is used to count the number of scrapes by HTTP status code.

This is the simplest query you can run:

promhttp_metric_handler_requests_total

The result in the table format will look like this:

promhttp_metric_handler_requests_total{code="200", instance="localhost:9090", job="prometheus"} 251
promhttp_metric_handler_requests_total{code="500", instance="localhost:9090", job="prometheus"} 0
promhttp_metric_handler_requests_total{code="503", instance="localhost:9090", job="prometheus"} 0

To see the same data for the last 24 hours, you can run the following query:

promhttp_metric_handler_requests_total[24h]

You will see the different values and their corresponding timestamps for each sample from the time series during the last 24 hours. The @ symbol is used to separate the value from the timestamp:

promhttp_metric_handler_requests_total{code="200", ..} 0 @1721038633.617
promhttp_metric_handler_requests_total{code="200", ..} 1 @1721038648.621
promhttp_metric_handler_requests_total{code="200", ..} 2 @1721038663.621
[..]
[..]
promhttp_metric_handler_requests_total{code="500",