Accelerating Developer Productivity via GitOps

What is GitOps?

GitOps is a methodology by which Kubernetes clusters can be managed via declarative manifests to enforce self-healing and self-reconciliation towards your desired state.

Compared to traditional CI/CD pipelines, GitOps follows a pull vs. push model. This means developers and operators do not need to invoke a pipeline to push changes into the cluster. Developers simply need to update their Kubernetes manifests in source control and the GitOps controllers running on the cluster will pull the changes, and apply the desired state. Therefore, Git becomes the single-source of truth for your environment.

Why implement GitOps?

In the past 11 years in industry, I have observed the benefits and pitfalls of numerous CI/CD systems from TeamCity to Jenkins to Gitlab and others. One common pattern I’ve viewed across organizations is the notion of shared CI/CD infrastructure. One or a few build servers are shared by dozens of teams which often result in resource contention on the server side, intermittent networking issues, frequent maintenance outages, which together serve as a bottleneck for development teams when they are not able to push builds. There are of course numerous benefits with these systems, but there have to be better ways.

Too often teams are regressed due to a dependency on a shared service.

GitOps allows us to scale horizontally the number of clusters as each cluster supports self-reconciliation and self-healing.

Architectural View

There are many technologies which support deploying a GitOps workflow. FluxCD and ArgoCD are two of the most common tools used. For the purpose of this article, we’ll explore GitOps through FluxCD.

As mentioned earlier, each GitOps enabled cluster runs a set of controllers responsible for reconciling cluster state from Git. These controllers are capable of reconciling from different repositories (e.g. Gitlab, Github, Bitbucket, etc.) and different manifests (e.g Kubernetes, Helm etc).

Below is a high-level architecture of these controllers in FluxCD:

Defined above are the following controllers, each of which are pods deployed as part of a Deployment resource:

• Source Controller: responsible for continuously pulling the manifests from the configured source repository and branch. The controller can be configured to use an HTTP egress proxy to communicate with these repositories if needed.
• Kustomize Controller: responsible for performing a kustomize build on the Kubernetes manifests retrieved by the Source Controller and apply them to the cluster through communication with the Kubernetes API Server. Kustomize is a tool which allows you to declaratively define Kubernetes manifests and template them through interpolation. More information on this will be linked below.
• Helm Controller: responsible for reconciling HelmRepository and HelmChart resources from source and deploying them onto the cluster. Through this controller, you can create a helm deployment manifest with a values file just as you would normally do. The HelmController will then be responsible for fetching the helm repositories. You can configure the source of the chart to be pulled from an upstream helm repository via the HelmRepositoryresource or the local Git repository through a GitRepository resource.

Git Repository Structure

You have the flexibility to define how you want to structure your Git repositories which are used as the source of truth in your GitOps workflow.

Below are a few practices you could employ:

• Define 1 Git Repository per 1 Kubernetes cluster
• Define N Kubernetes Clusters per 1 Git Repository

The first model is simpler if you have many K8s manifests and it will be easier to manage. The second model can work if your K8s manifest footprint per cluster is small. For the purpose of this article, we’ll stick with the first model.

Below is how you can structure your repository:

The kustomize.yaml file at the top is your root kustomization file which will build all of the resources define in your repository. Below is how you can define this file:

Notice above we have a set of -sync.yaml files defined. These are where we define how often we need to synchronize our cluster with the sources we’ve defined along with the path and repository where the sources are, as well as where various other advanced configurations we can apply towards our kustomization manifests in Flux, such as var substitution .

Below is an example of our app-sync.yaml manifest:

Within our app folder, we can continue to define our native Kubernetes manifests for our app, such as a Deploymentresource, a Service resource to expose it, and a ConfigMap for configuration. I won’t get into those definitions as there are various resources on the web and they have been widely documented.

Most importantly, we will define a GitRepository resource so the Flux source controller knows where to pull sources for reconciliation:

Bootstrapping our Cluster with Flux

With our GitRepository defined, you can bootstrap your cluster using the following command. Before you do that, ensure you have the flux CLI installed:

$ brew install --cask flux

Bootstrap command:

Following execution, flux will be bootstrapped on your cluster and configured to use the repository you define here. Additionally, all controllers will be deployed into the flux-system namespace. The flux CLI is also a great resource to explore as there are many commands you can use to interrogate the status of your cluster:

Check reconciliation statuses:

flux get kustomization -A

Suspend a reconciliation:

flux suspend kustomization app-sync

flux suspend helmrelease prometheus

Resume reconciliations:

flux resume kustomization app-sync

In conclusion, GitOps is a model you can use to accelerate developer productivity, deployment of your applications, and scales K8s clusters horizontally across your organization. Linked below are some additional resources to explore.

In later articles, I’ll dive in further with a hands-on-example.

The article represents my views only, and not reflective of any organization or company.

Resources

• FluxCD Documentation
• ArgoCD Documentation
• Kustomize Documentation