Cloud Native CI/CD with GitLab

Concurrency, Limits, and Request Concurrency

GitLab runners are by definition isolated processes (a container is a process) that handle job execution requests from GitLab CI. They work by picking up jobs through the GitLab coordinator API and running them according to predefined configurations. The runners can create multiple sub-processes, known as "executors," to run jobs concurrently.

GitLab provides configuration options to control the number of concurrent jobs that a runner can handle.

Example:

gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "$GITLAB_RUNNER_TOKEN"
rbac:
  create: true  
serviceAccount:
  create: true  
metrics:
  enabled: false  
  serviceMonitor:
    enabled: false  
runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "python:3.12"
  privileged: true  
  tags: 
    - kubernetes
concurrent: 20

This setting limits how many jobs can run concurrently across all runners on a given host. For example, if concurrent = 6, then no more than 6 jobs can run at the same time across all configured runners.

Another option is to limit the number of concurrent jobs per runner. This can be done by setting the limit option in the runner configuration. Here is an example:

gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "$GITLAB_RUNNER_TOKEN"
rbac:
  create: true  
serviceAccount:
  create: true  
metrics:
  enabled: false  
  serviceMonitor:
    enabled: false  
runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "python:3.12"
        limit = 5
  privileged: true  
  tags: 
    - kubernetes        

This setting specifies how many jobs a particular runner can handle concurrently. If you have multiple runners on the same machine, each runner's limit can be set separately. For instance, if limit = 5, that specific runner can run up to 5 jobs simultaneously. In other words, the number of sub-processes that the runner can create is limited to 5.

Finally, we have the request_concurrency option. This controls how many job requests a runner can process from the GitLab CI job queue at the same time. For example, request_concurrency = 3 means that the runner can process up to 3 job requests at the same time.

gitlabUrl