GitOps the Hard Way, with Argo CD

Goal: A Live Environment for Every Branch Push

Imagine you are a developer working on a new feature. You create a branch, write some code, and push the changes. Your pipeline builds, tests, lints, and runs other quality checks, but nothing beats seeing your changes deployed to a live environment where you can test them, share them, and iterate.

In cloud-native environments, developers are expected to be more autonomous and responsible for their code. "You build it, you run it" is a common mantra. Reaching that autonomy without automation, and without something close to a platform-as-a-service experience, is not realistic.

Argo CD is the deploy half of that automation. Coupled with GitLab CI/CD (or an alternative like GitHub Actions), you can build a continuous deployment pipeline that deploys your application whenever you create a branch or push to one. When you update your code, Argo CD updates the application in the target environment.

After a branch is merged elsewhere, a new environment can be deployed so you can test changes alongside other features. This is the value of multi-branch CI/CD. GitLab CI is one way to drive it.

GitLab CI uses a file called .gitlab-ci.yml to define a pipeline. It is a YAML file with a series of stages and jobs. It can be used to build, test, and deploy. Whenever there is a commit to a branch, GitLab CI runs the pipeline defined in that file.

In this chapter, we walk through the details. We use the same todo application from previous chapters. Docker containerizes the application, Helm packages it, and Kubernetes deploys it. GitLab CI is the orchestrator: it builds the Docker image, creates the Helm package, pushes both to the GitLab Container Registry and Package Registry respectively, and updates the Argo CD Application whenever there is a new commit.

There is no single way to do this. You will find opinionated choices. This is a prototype with one purpose: learning. Use it, change it, adapt it. Since it is impossible to adapt one example to every scenario, we keep it simple and focused on the main goal: let developers see their changes deployed in a live environment as soon as they push their code.

The steps the pipeline automates:

• Build a Docker image for the application.
• Push it to the GitLab Container Registry.
• Package a Helm chart that represents the Kubernetes deployment of the application.
• Push the chart to the GitLab Helm Package Registry.
• Update the Argo CD Application manifest to deploy the new version of the application to our Kubernetes cluster.

When deploying, we create a new namespace, a new Helm release, and a new Argo CD Application for each branch. That is why we use variables in the Argo CD Application manifest to represent the branch name and the target revision.

We are going to start afresh, so let's clean up the environment:

argocd app delete todo-app --yes

Here is the template we will use to create the Argo CD Application manifest:

cat <<'EOF' > $HOME/todo/app/manifests/app-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  # A unique name for the application
  name: $ARGOCD_APP_NAME
  namespace: argocd
spec:
  destination:
    # The namespace where the application will be deployed
    namespace: $HELM_NAMESPACE
    server: https://kubernetes.default.svc
  project: default
  source:
    chart: todo
    # The Helm repository URL
    repoURL: $HELM_REPO_URL
    # The target revision of the Helm chart
    targetRevision: $HELM_TARGET_REVISION
    helm:
      valueFiles:
      - values.yaml
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
EOF

(i) Since we need to create a new namespace for each branch, the Application manifest uses CreateNamespace=true in syncOptions. Argo CD creates the namespace if it does not already exist.

Let's also remove the files we used in previous chapters:

rm $HOME/todo/app/manifests/app-helm.yaml
rm $HOME/todo/app/manifests/app-plain.yaml
rm $HOME/todo/app/manifests/app-helm-with-repo.yaml

We almost have everything we need to start the CI/CD pipeline, except the Secret for the GitLab Container Registry.

In previous chapters, we created the Secret manually and applied it with:

kubectl apply -f $HOME/todo/app/manifests/registry-secret.yaml

There are two problems with this approach:

• The Secret is not managed by Helm.
• The Secret is only deployed in the default namespace where we used to deploy the todo application, and a Secret is namespaced.

Now that we deploy to multiple namespaces, we need a Secret in each namespace, otherwise pulling the image from the registry will fail. It is also better if the Secret is managed by Helm.

The fix is to let the chart create the Secret. When the Secret is a chart template, Helm renders it into the same namespace it installs the release into, so every branch namespace gets its own copy without any manual step.

Add a new template file to the chart at manifests/helm/todo/templates/registry-secret.yaml. Helm renders every file under templates/, so creating the file is all the wiring it needs:

cat <<'EOF' > $HOME/todo/app/manifests/helm/todo/templates/registry-secret.yaml
{{- if .Values.registry.create }}
apiVersion: v1
kind: Secret
metadata:
  # Reuse the name the pod already references in .Values.imagePullSecrets
  name: {{ (index .Values.imagePullSecrets 0).name }}
  # Render into whatever namespace Helm installs the release into
  namespace: {{ .Release.Namespace }}
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: {{ printf `{"auths":{"%s":{"username":"%s","password":"%s","auth":"%s"}}}` .Values.registry.server .Values.registry.username .Values.registry.password (printf "%s:%s" .Values.registry.username .Values.registry.password | b64enc) | b64enc }}
{{- end }}
EOF

Three parts of this template are worth explaining.

{{ .Release.Namespace }} is what makes the Secret multi-namespace. Helm sets it to the namespace of the release, so the same chart installed into main, feature-new, or any other branch namespace creates the Secret in that namespace.

{{ (index .Values.imagePullSecrets 0).name }} takes the name of the first entry in the imagePullSecrets list we already have in values.yaml. The Secret the chart creates and the Secret the pod references now come from a single value, so they cannot drift apart.

The .dockerconfigjson line encodes twice on purpose. The inner auth field is username:password encoded with b64enc, which is the format Docker expects. The whole JSON document is then encoded again because a Secret of type kubernetes.io/dockerconfigjson stores its .dockerconfigjson value base64-encoded.

This template is not specific to the todo chart. You can copy it into any of your Helm charts, as long as that chart's values.yaml keeps the same registry block and an imagePullSecrets list. The template depends on those value names, not on anything else in the chart.

Indeed, the template reads 4 keys under registry that do not exist in our values.yaml yet:

registry:
  # Toggle for the secret template; set to false to skip it
  create: true
  server: registry.gitlab.com
  # Filled at deploy time, left empty in the committed file
  username: ""
  password: ""

create and server carry real values and stay in the file. username and password are intentionally empty: committing registry credentials to Git would expose them to anyone who can read the repository. The pipeline supplies the real values at deploy time through the Argo CD Application's source.helm.parameters.

This is the code to run to create the new values file:

cat < $HOME/todo/app/manifests/helm/todo/values.yaml
replicaCount: 2
registry:
  create: true
  server: registry.gitlab.com  
  username: ""
  password: ""
image:
  repository: registry.gitlab.com/learning8960812/todo
  tag: v2
  pullPolicy: IfNotPresent
service:
  type: NodePort
  port: 5000
resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi
imagePullSecrets:
  - name: gitlab-registry
readinessProbe:
  httpGet:
    path: /tasks
    port: 5000
  initialDelaySeconds: 5
  periodSeconds: 10
autoscaling:
  enabled: false  
serviceAccount:
  create: false
ingress:
  enabled: false
httpRoute:
  enabled: false
EOF