Building User-Friendly Grafana Dashboard

We've already seen how to build custom, but basic, dashboards or import pre-built ones from the dashboard repository. In this section, we'll build a more advanced, user-friendly dashboard. We will use the Blackbox Exporter metrics as an example, but the same principles apply to any other metrics whatsoever.

Here is what we want to achieve: create a dashboard that dynamically shows probe_http_duration_seconds for the http_2xx module based on the host being probed (google.com, facebook.com, twitter.com, etc.). The user should be able to select one or more hosts and a particular phase (connect, processing, resolve, tls, transfer).

We are using google.com, facebook.com, and twitter.com as examples here, but you can add any target URL you want to monitor.

ℹ️ probe_http_duration_seconds is a gauge that shows the duration of the HTTP request by phase, summed over all redirects. A typical HTTP request has the following phases: connect (the time spent establishing the connection), resolve (the time spent resolving the DNS), tls (the time spent performing the TLS handshake), processing (the time spent waiting for the server to send data), and transfer (the time spent receiving the response data).

First of all, we need to make sure that the Blackbox Exporter is configured correctly. Run the following command to restore the initial configuration:

cat < /etc/prometheus/blackbox.yml && \
docker restart blackbox_exporter
modules:
  http_2xx:
    prober: http
    http:
      preferred_ip_protocol: "ip4"
EOF

Then, use this command to configure Prometheus:

cat < /etc/prometheus/prometheus.yml && \
kill -HUP $(pgrep prometheus)
scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - google.com
        - facebook.com
        - twitter.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115
EOF

The metric that we want to visualize is probe_http_duration_seconds. You can test it by running the following curl command on the monitoring host:

curl -s "http://localhost:9115/probe?target=google.com&module=http_2xx" | \
grep probe_http_duration_seconds | \
grep -v "#"

The output should look like this:

probe_http_duration_seconds{phase="connect"} 
probe_http_duration_seconds{phase="processing"} 
probe_http_duration_seconds{phase="resolve"} 
probe_http_duration_seconds{phase="tls"} 
probe_http_duration_seconds{phase="transfer"} 

For twitter.com, facebook.com, and google.com, the metrics are the same. The only difference is the value of the instance label and the target URL. Notice that this metric is split into different phases: connect, processing, resolve, tls, and transfer. Our goal is to visualize the sum of these phases for each target URL (the instance label). For this purpose, we can use one of the following options:

• probe_duration_seconds, which is the total duration of the probe request. This could include some overhead that we may not want to consider, but it's okay (the extra overhead is usually extremely low).
• Sum of probe_http_duration_seconds over all phases. This is more accurate, since it only considers the HTTP request duration.

We are going to consider the second option. The PromQL query for it is as follows:

sum by (instance) (
  probe_http_duration_seconds
)

You can test this in the Prometheus UI, and it will show you a result per target URL. Here is an example:

{instance="twitter.com"} 0.37723951
{instance="facebook.com"} 0.465827366
{instance="google.com"} 0.068915492

To show the result for a specific target URL, you can use the instance