Through deployment of a GitOps model, developers can delivery self-reconciliation and self-healing across Kubernetes clusters, without using a CI/CD system, and scale horizontally at massive scale.
Below I provide a walkthrough of GitOps from a 50,000 ft. view.
GitOps is a methodology by which Kubernetes clusters can be managed via declarative manifests to enforce self-healing and self-reconciliation towards your desired state.
Compared to traditional CI/CD pipelines, GitOps follows a pull vs. push model. This means developers and operators do not need to invoke a pipeline to push changes into the cluster. Developers simply need to update their Kubernetes manifests in source control and the GitOps controllers running on the cluster will pull the changes, and apply the desired state. Therefore, Git becomes the single-source of truth for your environment.
In the past 11 years in industry, I have observed the benefits and pitfalls of numerous CI/CD systems from TeamCity to Jenkins to Gitlab and others. One common pattern I’ve viewed across organizations is the notion of shared CI/CD infrastructure. One or a few build servers are shared by dozens of teams which often result in resource contention on the server side, intermittent networking issues, frequent maintenance outages, which together serve as a bottleneck for development teams when they are not able to push builds. There are of course numerous benefits with these systems, but there have to be better ways.
Too often teams are regressed due to a dependency on a shared service.
GitOps allows us to scale horizontally the number of clusters as each cluster supports self-reconciliation and self-healing.
There are many technologies which support deploying a GitOps workflow. FluxCD and ArgoCD are two of the most common tools used. For the purpose of this article, we’ll explore GitOps through FluxCD.
As mentioned earlier, each GitOps enabled cluster runs a set of controllers responsible for reconciling cluster state from Git. These controllers are capable of reconciling from different repositories (e.g. Gitlab, Github, Bitbucket, etc.) and different manifests (e.g Kubernetes, Helm etc).
Below is a high-level architecture of these controllers in FluxCD:
Defined above are the following controllers, each of which are pods deployed as part of a
kustomizebuild on the Kubernetes manifests retrieved by the
Source Controllerand apply them to the cluster through communication with the Kubernetes API Server. Kustomize is a tool which allows you to declaratively define Kubernetes manifests and template them through interpolation. More information on this will be linked below.
HelmChartresources from source and deploying them onto the cluster. Through this controller, you can create a helm deployment manifest with a values file just as you would normally do. The
HelmControllerwill then be responsible for fetching the helm repositories. You can configure the source of the chart to be pulled from an upstream helm repository via the
HelmRepositoryresource or the local Git repository through a
You have the flexibility to define how you want to structure your Git repositories which are used as the source of truth in your GitOps workflow.
Below are a few practices you could employ:
The first model is simpler if you have many K8s manifests and it will be easier to manage. The second model can work if your K8s manifest footprint per cluster is small. For the purpose of this article, we’ll stick with the first model.
Below is how you can structure your repository:
kustomize.yaml file at the top is your root kustomization file which will build all of the resources define in your repository. Below is how you can define this file:
Notice above we have a set of
-sync.yaml files defined. These are where we define how often we need to synchronize our cluster with the sources we’ve defined along with the path and repository where the sources are, as well as where various other advanced configurations we can apply towards our kustomization manifests in Flux, such as
var substitution .
Below is an example of our
app folder, we can continue to define our native Kubernetes manifests for our app, such as a
Service resource to expose it, and a
ConfigMap for configuration. I won’t get into those definitions as there are various resources on the web and they have been widely documented.
Most importantly, we will define a
GitRepository resource so the Flux source controller knows where to pull sources for reconciliation:
With our GitRepository defined, you can bootstrap your cluster using the following command. Before you do that, ensure you have the
flux CLI installed:
$ brew install --cask flux
Following execution, flux will be bootstrapped on your cluster and configured to use the repository you define here. Additionally, all controllers will be deployed into the
flux-system namespace. The
flux CLI is also a great resource to explore as there are many commands you can use to interrogate the status of your cluster:
Check reconciliation statuses:
flux get kustomization -A
Suspend a reconciliation:
flux suspend kustomization app-sync
flux suspend helmrelease prometheus
flux resume kustomization app-sync
In conclusion, GitOps is a model you can use to accelerate developer productivity, deployment of your applications, and scales K8s clusters horizontally across your organization. Linked below are some additional resources to explore.
In later articles, I’ll dive in further with a hands-on-example.
The article represents my views only, and not reflective of any organization or company.