Putting It All Together — From Practices to Pipelines
Building a DevSecOps Pipeline
To show a practical example of how to implement the above concepts, we will create a GitLab CI/CD pipeline that uses some of the tools we have seen in previous chapters. This pipeline will be triggered on every commit and will run the following security checks:
- Verify commit signature
- Build a Python virtual environment and install dependencies
- Scan for secrets using TruffleHog
- Scan for vulnerable dependencies using OWASP Dependency-Check
- Scan for Python security issues using Bandit
- Lint the Dockerfile using Hadolint
- Build the Docker image and push it to the GitLab registry
- Scan the Docker image for vulnerabilities using Trivy
- Scan the Terraform files for security issues using Checkov
- Lint the Kubernetes manifests using KubeLinter
- Generate a Software Bill of Materials (SBOM) using Syft
Jobs are configured to allow failure, meaning that the pipeline will not stop if a job fails. This is useful for testing purposes, but in a production environment, you should set the allow_failure option to false. Some jobs are configured to create reports in JSON format and publish them as artifacts. This allows you to download the reports after the pipeline finishes. You can customize the pipeline to suit your needs, but this example should give you a good starting point.
After following the steps in this chapter, you should be able to have the following pipeline running in your GitLab project:
GitLab CI/CD pipeline example
Before starting, since the pipeline will be blocking all unsigned commits, we can implement a GPG keyring to verify the commit signatures.
To keep things simple, we will export the GPG keys of the developers who will be working on the project. This can be done by running the following commands:
# Export each key
gpg --export --armor KEYID1 > dev1.asc
gpg --export --armor KEYID2 > dev2.asc
gpg --export --armor KEYID3 > dev3.asc
# Combine them into one and encode it in base64
cat dev1.asc dev2.asc dev3.asc > devs.key
# Base64 encode the keyring for GitLab CI/CD
cat devs.key | base64 -w0 > devs.key.base64
Create a new secret variable in GitLab CI/CD called GPG_KEYRING and paste the base64 encoded keyring. To do this, go to your GitLab project, navigate to Settings > CI/CD > Variables, and add the new variable. Make sure to set the Masked option to true to prevent it from being exposed in the logs.
Use the same process to export the variable NVD_API_KEY for the OWASP Dependency-Check tool. This is the free API key that you can get from the NVD website. We already have one, but feel free to create a new one if you lost the first one.
In the following sections, we will only focus on the menu microservice, but you can apply the same principles to the other microservices.
Verifying the Commit Signature
Before running any job, it's important to ensure the commit is properly signed using GPG. This job uses git verify-commit and fails if the commit is unsigned or invalid. It lays the foundation for the rest of the pipeline.
cat <<'EOF' > $HOME/RestQR/.gitlab-ci.yml
# Verify commit signature
# If the commit is not signed, the job will fail
verify_commit_signature:
allow_failure: true # Do not fail the whole pipeline if this fails (can be set to false for stricter policies)
image: alpine:latest
before_script:
- apk add --no-cache git gnupg # Install Git and GPG
- echo "$GPG_KEYRING" | base64 -d > /tmp/devs.key # Decode and save GPG keyring
- gpg --import /tmp/devs.key # Import GPG keys
script:
- echo "Verifying commit signature for $CI_COMMIT_SHA"
- |
if git verify-commit "$CI_COMMIT_SHA"; then
echo "Commit is GPG signed and verified."
else
echo "Commit is NOT signed or signature is invalid."
exit 1
fi
EOF
Building the Application
This job sets up a Python virtual environment and installs dependencies from the menu/requirements.txt file. The resulting environment is saved as an artifact for later use.
cat <<'EOF' >> $HOME/RestQR/.gitlab-ci.yml
# Set up a Python virtual environment and install project dependencies
build:
allow_failure: true
image: python:3.10
script:
- python -m venv venv # Create virtual environment
- source venv/bin/activate
- pip install --upgrade pip
- pip install -r menu/requirements.txt # Install required Python packages
artifacts:
name: "menu-env"
paths:
- venv/
needs: [verify_commit_signature]
EOF
Scanning for Secrets Using TruffleHog
This step scans the repository for any committed secrets using TruffleHog. The job only runs if the commit is verified.
cat <<'EOF' >> $HOME/RestQR/.gitlab-ci.yml
# Scan the repository for secrets using TruffleHog
trufflehog_scan:
allow_failure: true
image: alpine:latest
before_script:
- apk add --no-cache git curl jq # Install required tools
- curl -sSfL https://raw.githubusercontent.com/trufflesecurity/trufflehog/main/scripts/install.sh | sh -s -- -b /usr/local/bin # Install TruffleHog
script:
- trufflehog filesystem . --only-verified --json | tee trufflehog-results.json | jq
artifacts:
paths:
- trufflehog-results.json
when: always
needs: [verify_commit_signature]
EOF
Scanning for Vulnerable Dependencies Using OWASP Dependency-Check
This job analyzes project dependencies for known CVEs. It requires the commit to be signed and the environment to be built first.
cat <<'EOF' >> $HOME/RestQR/.gitlab-ci.yml
# Perform dependency vulnerability analysis using OWASP Dependency-Check
owasp_dependency_check:
allow_failure: true
image:
name: registry.gitlab.com/gitlab-ci-utils/docker-dependency-check:4.4.0
entrypoint: [""]
script:
- >
/usr/share/dependency-check/bin/dependency-check.sh --scan "./" --format ALL
--project "$CI_PROJECT_NAME" --nvdApiKey $NVD_API_KEY --failOnCVSS 0 # Fail if any vulnerability found
artifacts:
when: always
paths:
- "./dependency-check-report.html"
- "./dependency-check-report.json"
needs: [verify_commit_signature, build]
EOF
Scanning for Python Security Issues Using Bandit
This step scans Python code for security issues using Bandit and stores the results in an HTML report.
cat <<'EOF' >> $HOME/RestQR/.gitlab-ci.yml
# Scan Python code with Bandit for security issues
bandit_scan:
allow_failure: true
image: python:3.10
script:
- source venv/bin/activate
- pip install bandit==1.8.3
- bandit -r . -f html -o bandit-report.html || true # Run Bandit and save HTML report
artifacts:
paths:
- bandit-report.html
needs: [verify_commit_signature, build]
EOF
Linting the Dockerfile Using Hadolint
This job ensures that Dockerfiles follow best practices by running Hadolint and saving results.
cat <<'EOF' >> $HOME/RestQR/.gitlab-ci.yml
# Lint Dockerfile with Hadolint
lint_dockerfile:
allow_failure: true
image: hadolint/hadolint:v2.12.0-debian
script:
- hadolint -f json menu/Dockerfile > hadolint-report.json || true # Save linting results as JSON
artifacts:
name: "$CI_JOB_NAME artifacts from $CI_PROJECT_NAME DevSecOps in Practice
A Hands-On Guide to Operationalizing DevSecOps at ScaleEnroll now to unlock current content and receive all future updates for free. Your purchase supports the author and fuels the creation of more exciting content. Act fast, as the price will rise as the course nears completion!