Cloud Native CI/CD with GitLab

Concurrency, Limits, and Request Concurrency

GitLab runners launch jobs in parallel in containers to maximize resource utilization and reduce job wait times. However, running too many jobs concurrently can lead to resource contention, such as network or disk I/O timeouts. To avoid such issues, GitLab provides configuration options to control the number of concurrent jobs that a runner can handle.

Example: This setting limits how many jobs can run concurrently across all runners on a given host. For example, if concurrent = 6, then no more than 6 jobs can run at the same time across all configured runners. You can change the concurrent option in the GitLab Runner Helm chart values file like this:

gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "$GITLAB_RUNNER_TOKEN"
rbac:
  create: true
serviceAccount:
  create: true
metrics:
  enabled: false
  serviceMonitor:
    enabled: false
runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "python:3.12"
  privileged: true
  tags:
    - kubernetes
concurrent: 20

Another option is to limit the number of concurrent jobs per runner. This can be done by setting the limit option in the runner configuration. Here is an example:

gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "$GITLAB_RUNNER_TOKEN"
rbac:
  create: true
serviceAccount:
  create: true
metrics:
  enabled: false
  serviceMonitor:
    enabled: false
runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "python:3.12"
        limit = 5
  privileged: true
  tags:
    - kubernetes

This setting specifies how many jobs a particular runner can handle concurrently. If you have multiple runners on the same machine, each runner's limit can be set separately. For instance, if limit = 5, that specific runner can run up to 5 jobs simultaneously. In other words, the number of sub-processes that the runner can create is limited to 5.

Finally, we have the request_concurrency option. This controls how many job requests a runner can process from the GitLab CI job queue at the same time. For example, request_concurrency = 3 means that the runner can process up to 3 job requests at the same time.

gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "$GITLAB_RUNNER_TOKEN"
rbac: