Understanding Job Dependencies and Stageless Pipelines

Direct Acyclic Graph (DAG) Pipelines: Non-Sequential Faster Pipelines

41%

In GitLab adding dependencies between jobs is done using the needs keyword and in addition to its main purpose, it can also be used to speed up the pipeline. To understand this better, let's start with a first example that launches two jobs in parallel (build-1 and build-2) in the same stage:

stages:
  - build

build-1:
  stage: build
  script:
    - echo "Building the first part of the code..."

build-2:
  stage: build
  script:
    - echo "Building the second part of the code..."

By default, two jobs running in the same stage are executed in parallel. If you want to run build-2 after build-1 while keeping them in the same stage, you should use the needs keyword. This example demonstrates how to do it.

Run the following code to update the .gitlab-ci.yml file and trigger a new pipeline execution then navigate to the GitLab UI to see the outcome:

cat < $HOME/todo/app/.gitlab-ci.yml && \
cd $HOME/todo/app && \
git add . && \
git commit -m "Add dependencies between jobs" && \
git push origin main
stages:
  - build

build-1:
  stage: build
  script:
    - echo "Building the first part of the code..."

build-2:
  stage: build
  script:
    - echo "Building the second part of the code..."
  needs:
    - build-1
EOF

The build-2 job runs after the build-1 job even if they are in the same stage.

Now, let's take this example where we have three jobs in three different stages:

stages:
  - build-1
  - build-2
  - build-3

build-1:
  stage: build-1
  script:
    - echo "Building the first part of the code..."

build-2:
  stage: build-2
  script:
    - echo "Building the second part of the code..."

build-3:
  stage: build-3
  script:
    - echo "Building the third part of the code..."

By default, jobs in different stages are executed sequentially:

build-1 runs first
followed by build-2
and finally build-3

In some cases, we may need to launch a job or a stage as soon as possible. This is where the needs may help us to speed up the pipeline.

In the following pipeline, the regular behavior is to run the all the build-x jobs sequentially followed by the my-job job.

stages:
  - build-1
  - build-2
  - build-3
  - build-4
  - build-5
  - my-stage

build-1:
  stage: build-1
  script:
    - echo "Building the first part of the code..."

build-2:
  stage: build-2
  script:
    - echo "Building the second part of the code..."

build-3:
  stage: build-3
  script:
    - echo "Building the third part of the code..."

build-4:
  stage: build-4
  script:
    - echo "Building the fourth part of the code..."

build-5:
  stage: build-5
  script:
    - echo "Building the fifth part of the code..."

my-job:
  stage: my-stage
  script:
    - echo "Am I the last job?"

Here's a scheme of the default execution order:

build-1
   |
   +-----> build-2
           |
           +-----> build-3
                   |
                   +-----> build-4
                           |
                           +-----> build-5
                                   |
                                   +-----> my-

Cloud Native CI/CD with GitLab

From Commit to Production Ready

Enroll now to unlock all content and receive all future updates for free.

Unlock now $29.99 Learn More

Previous Next