Observability with Prometheus and Grafana

Configuring Prometheus

As seen previously, the default configuration file for Prometheus is located at /etc/prometheus/prometheus.yml. We are going to start with a simple configuration file that scrapes the Prometheus server itself. This is typically the first configuration you need to use when you run a Prometheus instance.

You can execute the following command to update the configuration file:

cat < /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
EOF

This configuration file tells Prometheus to scrape itself every 15 seconds. The job_name is prometheus and the target is localhost:9090.

You can reload the configuration file by sending a SIGHUP signal to the Prometheus process:

kill -HUP $(pgrep prometheus)

ℹ️ The kill command sends a signal to a process. The HUP signal (hang up) is used to reload the configuration file of a daemon (by convention). The pgrep command is used to get the process ID of a process by its name.

Alternatively, if you prefer to restart the service, you can do so by running the following command:

systemctl restart prometheus

You can also use the web server to do the same thing:

curl -X POST http://$(curl -s ifconfig.me):9090/-/reload

The latter is only available if you have enabled the --web.enable-lifecycle flag in the Prometheus service file.

Back to our configuration file. First of all, it's a YAML file that follows a specific structure. Here are the main sections:

• global is the global configuration for Prometheus.

• The scrape_interval is the interval at which Prometheus scrapes the targets.

• scrape_configs is the list of targets that Prometheus scrapes.

• job_name is the name of the job that we use to identify the targets.
• static_configs block is used to define the targets statically (as opposed to dynamically discovered targets).
  • targets is the list of targets that Prometheus scrapes. The format is hostname:port.

ℹ️ In YAML, indentation is important. It's recommended to use spaces (e.g., 2 or 4 spaces) instead of tabs for indentation.

Note that our file can also be written using this format:

global:
  scrape_interval: 15s

scrape_configs: