Cloud Native CI/CD with GitLab

Monitoring GitLab Runners

Most tools and services that are part of the cloud-native ecosystem provide monitoring capabilities to help you track the performance and health of your applications and infrastructure. GitLab Runner is no exception. GitLab Runner is Prometheus-compatible, which means you can monitor the runner's performance and resource usage using Prometheus. By default, GitLab Runner exposes metrics on the /metrics endpoint, which can be scraped by Prometheus. To enable monitoring, you need to configure Prometheus to scrape the runner and the runner to expose metrics if it is not already enabled.

ℹ️ Prometheus is an open-source monitoring and alerting toolkit that is widely used in the cloud-native ecosystem. It collects metrics from various sources, stores them in a time-series database, and provides a query language to analyze the data and create alerts based on predefined rules.

Prometheus can be installed as a standalone service or as part of a Kubernetes cluster. However, it's more common to use it to monitor Kubernetes clusters and containerized applications running on Kubernetes. In this section, we will focus on monitoring GitLab Runner using Prometheus in a Kubernetes environment. You can apply similar steps to other environments like standalone servers or cloud instances with minor modifications - the core concepts remain the same.

This section assumes you have the K3s cluster seen in the previous section set up and running. If you don't have a K3s cluster, please refer to the previous section.

Since Prometheus will be deployed to the namespace monitoring, let's prepare this at the beginning:

# Create the monitoring namespace
kubectl create namespace monitoring

Back to our GitLab runner, we should configure it to expose Prometheus metrics. We can do this by updating the Helm values file. Make sure you exported the runner token as an environment variable (GITLAB_RUNNER_TOKEN) then run the following commands to update the values file for the Helm chart:

cat < $HOME/todo/gitlab-runner/helm/values.yaml
# We already have the configuration 
# for the GitLab Runner in the values file.
gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "$GITLAB_RUNNER_TOKEN"
rbac:  
  create: true  
serviceAccount:  
  create: true  
runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "python:3.12"
  privileged: true  

# This is what we're updating
# Metrics configuration.
metrics:  
  # Enable gitlab-runner metrics
  enabled: true
  serviceMonitor:    
    # Deploy a ServiceMonitor in the monitoring namespace
    namespace: monitoring
    # Enable service monitor  
    enabled: true 
    labels:
      app: gitlab-runner
      # IMPORTANT notes: 
      # 1. Adding the release is mandatory 
      # for the ServiceMonitor to work
      # 2. The release label must match 
      # the release name of the Prometheus Operator
      release: prometheus-operator
service:
  enabled: true
  type: ClusterIP
EOF

As you can see, we enabled metrics and created a ServiceMonitor in the monitoring namespace. We also added labels to the ServiceMonitor to match the Prometheus Operator release name. This can be confusing at first, because we didn't install Prometheus yet. We will do that in the next steps and I'll get back to this point (the release label) later.

ℹ️ The ServiceMonitor resource is part of the Prometheus Operator. It is used to define the set of services that should be monitored by Prometheus. The ServiceMonitor resource specifies the service to monitor, the namespace in which the service resides, and the labels to match the service. An Operator, in Kubernetes, is a method of packaging, deploying, and managing a Kubernetes application. The Prometheus Operator is what we are going to use to deploy Prometheus and manage the Prometheus instances.

Now, you can upgrade the GitLab Runner Helm chart to apply the changes:

helm upgrade \
--namespace default \
gitlab-runner \
-f $HOME/todo/gitlab-runner/helm/values.yaml \
gitlab/gitlab-runner

If everything is set up correctly, you should see Metrics server listening address=:9252 in the logs of the runner pod when you run the following command:

kubectl logs -l app=gitlab-runner

This means that the runner is now exposing metrics on ":9252/metrics".

If you don't see the same message (because the logs were truncated), you may see other messages that indicate the metrics are enabled (like Job Succeded, Updating Job, etc.).

ℹ️ By default, Prometheus scrapes metrics from the /metrics endpoint. The port number can be different depending on the configuration.

You can see these metrics by running the following command (wait a few seconds after the upgrade to let the runner pod start):