Observability with Prometheus and Grafana

Integrating the Blackbox Exporter with Prometheus

To integrate the Blackbox Exporter with Prometheus, you need to add a new job to the Prometheus configuration file. The job will scrape the metrics exposed by the Blackbox Exporter. The configuration should include the target URLs that you want to probe and the module that you want to use. Let's use the following configuration as a starting point:

cat < /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
      target: ['google.com']
    static_configs:
      - targets:
        - localhost:9115
EOF

By default, this is how the URL of the probe looks like:

http:///?module=&target=

Where:

• static_configs.targets is the address of the Blackbox Exporter.
• metrics_path is the path where the Blackbox Exporter exposes the metrics. By default, it is /metrics, but for probing, we use /probe.
• params.module is the value of the module query parameter in the probe URL. In this case, it is http_2xx.
• params.target is the value of the target query parameter in the probe URL. In this case, it is google.com.

If you put everything together, the final URL that Prometheus will scrape looks like this:

http://localhost:9115/probe?module=http_2xx&target=google.com

We should reload the Prometheus configuration:

kill -HUP $(pgrep prometheus)

You can now see the new targets added to Prometheus by visiting http://:9090/targets in your browser.

echo http://$(curl -s ifconfig.me):9090/targets

One of the problems with the current configuration is that the labels are not very descriptive. As you can see, the instance label is set to localhost:9115. Logically, this label should be set to the scraped instance, which is google.com in our case, and not the Blackbox Exporter address.

Also, if we want to use another URL in addition to google.com, we would have a single instance label for all metrics from the different URLs. This would make it difficult to make sense of the metrics.

Example: We want to add Facebook and Twitter to the blackbox job:

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
      target: ['google.com', 'facebook.com', 'twitter.com']
    static_configs:
      - targets:
        - localhost:9115

The exported metrics will look like this:

# Is this the result of the probe for google.com, facebook.com or twitter.com?
probe_http_duration_seconds{instance="localhost:9115",phase="connect"} 0.008696507000000001
# Same question here!
probe_http_duration_seconds{instance="localhost:9115",phase="connect"} 0.004812664198761231
# ...

The question is: which instance label corresponds to which target URL: google.com, facebook.com, or twitter.com?

A solution would be to use the following:

• Use the static_configs.targets section to define the target URLs (google.com, facebook.com, twitter.com).
• Copy the value of the targets label to the __param_target label.
• Copy the value of the __param_target label to the instance label. This will set the instance label to the target URL (google.com, facebook.com, twitter.com).
• Replace the value of the __address__ label with localhost:9115 (the address and port of the Blackbox Exporter).

This is how the new blackbox job configuration looks:

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        # Add the target URLs here
        - google.com
    relabel_configs:
      # Copy the value of the `targets` label to the `__param_target` label
      - source_labels: [__address__]
        target_label: __param_target
      # Copy the value of the `__param_target` label to the `instance` label
      - source_labels: [__param_target]
        target_label: instance
      # Replace the value of the `__address__` label with `localhost:9115`
      - target_label: __address__
        replacement: localhost:9115