48%

Direct Acyclic Graph (DAG) Pipelines: Non-Sequential Faster Pipelines

In GitLab adding dependencies between jobs is done using the needs keyword and in addition to its main purpose, it can also be used to speed up the pipeline. To understand this better, let's start with a first example that launches two jobs in parallel (job1 and job2):

stages:
  - build

build-1:
  stage: build
  script:
    - echo "Building the first part of the code..."

build-2:
  stage: build
  script:
    - echo "Building the second part of the code..."

By default, two jobs running in the same stage are executed in parallel. If you want to run build-2 after build-1 while keeping them in the same stage, you should use the needs keyword. This example demonstrates how to run the build-2 job after the build-1 job. Use the code below to update the .gitlab-ci.yml file and trigger the pipeline:

cat < $HOME/todo/app/.gitlab-ci.yml && \
cd $HOME/todo/app && \
git add . && \
git commit -m "Add dependencies between jobs" && \
git push origin main
stages:
  - build

build-1:
  stage: build
  script:
    - echo "Building the first part of the code..."

build-2:
  stage: build
  script:
    - echo "Building the second part of the code..."
  needs:
    - build-1
EOF

When you run the pipeline, you will see that the build-2 job runs after the build-1 job.

Now, let's take this example where we have three jobs in three different stages:

stages:
  - build-1
  - build-2
  - build-3

build-1:
  stage: build-1
  script:
    - echo "Building the first part of the code..."

build-2:
  stage: build-2
  script:
    - echo "Building the second part of the code..."

build-3:
  stage: build-3
  script:
    - echo "Building the third part of the code..."

By default, jobs in different stages are executed sequentially, however, in some cases, we may need to launch a job or a stage as soon as possible. This is where the needs may help us to speed up the pipeline. Let's see an example. In the following pipeline, the regular behavior is to run the all the build-x jobs sequentially followed by the my-job job.

stages:
  - build-1
  - build-2
  - build-3
  - build-4
  - build-5
  - my-stage

build-1:
  stage: build-1
  script:
    - echo "Building the first part of the code..."

build-2:
  stage: build-2
  script:
    - echo "Building the second part of the code..."

build-3:
  stage: build-3
  script:
    - echo "Building the third part of the code..."

build-4:
  stage: build-4
  script:
    - echo "Building the fourth part of the code..."

build-5:
  stage: build-5
  script:
    - echo "Building the fifth part of the code..."

my-job:
  stage: my-stage
  script:
    - echo "Am I the last job?"

However, we want to run the my-job job as soon as the build-1 job is completed. By using the needs keyword, we can achieve this. Use the code below to update the .gitlab-ci.yml file and trigger a new pipeline execution:

cat <$HOME/todo/app/.gitlab-ci.yml && \
cd $HOME/todo/app && \
git add . && \
git commit -m "Add needs keyword" && \
git push origin main

Cloud Native CI/CD with GitLab

From Commit to Production Ready

Enroll now to unlock all content and receive all future updates for free.

Unlock now ($29.99) Learn More

Previous Next