Kubernetes: How to Use Kustomize with Kubernetes

0_732-IMZDPe_PDMD4.jpg

Having used Kubernetes consistently at work, I had to find a tool that could make my work easier and much better. Let us get into Kustomize and see how to use it at our work.

Architecture

From their site, Kustomize has a very wide usage:

  • Kustomize helps customizing config files in a template free way.
  • Kustomize provides a number of handy methods like generators to make customization easier.
  • Kustomize uses patches to introduce environment specific changes on an already existing standard config file without disturbing it.

It’s a perfect tool to create environment based customizations to your k8s deployments. kustomize uses a concept of bases and overlays, where you define a base and then you create overlays which customize the configuration depending on your environment. There is a pretty cool diagram in their github:

And here is a simple directory hierarchy:

                ~/someApp
├── base
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── service.yaml
└── overlays
    ├── development
    │   ├── cpu_count.yaml
    │   ├── kustomization.yaml
    │   └── replica_count.yaml
    └── production
        ├── cpu_count.yaml
        ├── kustomization.yaml
        └── replica_count.yaml
            

It’s also a perfect tool to be added into your CI/CD pipeline. From Workflow for off the shelf applications, here is a nice overview:

So let’s run through a couple of examples to see how it works. As a starting point, I created the following directory structure:

                > mkdir -p {overlays/{dev,prod},base}
> touch {base/kustomization.yaml,overlays/{dev,prod}/kustomization.yaml}
> tree
.
├── base
│   └── kustomization.yaml
└── overlays
    ├── dev
    │   └── kustomization.yaml
    └── prod
        └── kustomization.yaml
4 directories, 3 files
            

Container Images

First let’s try out modifying the container image version (this example is covered in customizing). As an example let’s create a simple deployment manifest:

                > cat base/deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: the-deployment
  annotations:
    app: nginx
spec:
  template:
    spec:
      containers:
      - name: nginxapp
        image: nginx:1.7.9
            

And here is a simple kustomization to modify the tag:

                > cat base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yaml
images:
- name: nginx
            

Now to see how it gets parsed, we can use kubectl kustomize:

                > k kustomize base           
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    app: nginx
  name: the-deployment
spec:
  template:
    spec:
      containers:
      - image: nginx:1.8.0
        name: nginxapp
            

To actually apply it, we can run kubectl apply -k (vs kubectl apply -f):

                k apply -k base
deployment.apps/the-deployment created
            

That was pretty easy.

Patching

kustomize has two approaches to patching files: strategic merge patch and json 6902 patch.

Strategic Merge Patch

This is useful if you are making a lot of changes, so you can just provide a YAML file that looks like a k8s manifest file with all the resources you want to add/change. From the Patching multiple resources at once example, let’s inject a side car container into our deployment in our dev overlay. First let’s add a kustomization.yaml file into the dev overlay:

                > cat overlays/dev/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: deploy-patch.yaml
            

And now let’s add the patch file to specify the container we want to add:

                > cat overlays/dev/deploy-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: the-deployment
spec:
  template:
    spec:
      containers:
        - name: istio-proxy
          image: docker.io/istio/proxyv2
          args:
          - proxy
          - sidecar
            

And here is how the parsed manifest will end up looking like:

                > k kustomize overlays/dev           
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    app: nginx
  name: the-deployment
spec:
  template:
    spec:
      containers:
      - args:
        - proxy
        - sidecar
        image: docker.io/istio/proxyv2
        name: istio-proxy
      - image: nginx:1.8.0
        name: nginxapp
            

That looks good.

JSON 6902 Patch

This is good for adding/modifying one field at a time (also the parent path has to exist, so it doesn’t create sub fields, for more information check out this github feature request). Let’s say I want add a new annotation and to a serviceaccount, we could create the following patch:

                > cat overlays/dev/deploy-more-patch.yaml
- op: add
  path: /metadata/annotations/app.io~1owner
  value: "me"
- op: add
  path: /spec/template/spec/serviceAccountName
  value: app
            

Notice that I had to use ~1 to specify a forward slash / (this is covered in this github issue). And here is update kustomization.yaml to include the new patch file. Notice that I also added a target to specify which resource to modify:

                > cat overlays/dev/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: deploy-patch.yaml
- path: deploy-more-patch.yaml
  target:
    kind: Deployment
            

And now here is the resulted manifest:

                > k kustomize overlays/dev                
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    app: nginx
    app.io/owner: me
  name: the-deployment
spec:
  template:
    spec:
      containers:
      - args:
        - proxy
        - sidecar
        image: docker.io/istio/proxyv2
        name: istio-proxy
      - image: nginx:1.8.0
        name: nginxapp
      serviceAccountName: app
            

yay, that worked out as well.

ConfigGenerator

Let’s cover one more example, the configMapGenerator, this supports multiple approaches as well. I will cover two.

Using Literals to Specify Configs as Strings

There is also a nice example at ConfigMap generation and rolling updates. Let’s say I create the following in my base:

                > cat base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yaml
images:
- name: nginx
  newTag: 1.8.0
configMapGenerator:
- name: config
  literals:
    - common="Common Variable"
            

Now at the overlay kustomization:

                > cat overlays/dev/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: deploy-patch.yaml
- path: deploy-more-patch.yaml
  target:
    kind: Deployment
configMapGenerator:
- name: config
  behavior: merge
  literals:
    - custom="My Custom Config"
            

Now I can modify the the deployment in the base to add the common configMap:

                > cat base/deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: the-deployment
  annotations:
    app: nginx
spec:
  template:
    spec:
      containers:
      - name: nginxapp
        image: nginx:1.7.9
        env:
        - name: COMMON
          valueFrom:
            configMapKeyRef:
              name: config
              key: common
            

And then I added a patch in the overlay to append the environment variable (since I my sidecar proxy patch from before added up adding the container first, I had to basically append it to the second container of my deployment… 0 is the first and 1 is the second):

                > cat overlays/dev/deploy-more-patch.yaml 
- op: add
  path: /metadata/annotations/app.io~1owner
  value: "me"
- op: add
  path: /spec/template/spec/serviceAccountName
  value: app
- op: add
  path: /spec/template/spec/containers/1/env/-
  value:
      name: CUSTOM
      valueFrom:
        configMapKeyRef:
          name: config
          key: custom
            

Parsing the config, here is what it creates:

                > k kustomize overlays/dev
apiVersion: v1
data:
  common: Common Variable
  custom: My Custom Config
kind: ConfigMap
metadata:
  name: config-k96cb96f75
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    app: nginx
    app.io/owner: me
  name: the-deployment
spec:
  template:
    spec:
      containers:
      - args:
        - proxy
        - sidecar
        image: docker.io/istio/proxyv2
        name: istio-proxy
      - env:
        - name: COMMON
          valueFrom:
            configMapKeyRef:
              key: common
              name: config-k96cb96f75
        - name: CUSTOM
          valueFrom:
            configMapKeyRef:
              key: custom
              name: config-k96cb96f75
        image: nginx:1.8.0
        name: nginxapp
      serviceAccountName: app
            

We can see it creates a dynamic configMap name and adds it to our container accordingly… that is pretty cool :)

Using Files to Specify Configs

Another approach is to provide configuration files in the configMaps. Each file ends up being a new data entry in the configMap. Another cool example at Demo: combining config data from devops and developers. Let’s add it to our prod overlay. First let’s modify our base and add a new configMapGenerator:

                > cat base/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yaml
images:
- name: nginx
  newTag: 1.8.0
configMapGenerator:
- name: config
  literals:
    - common="Common Variable"
- name: file-config
  files:
    - common.properties
            

Now let’s add it to our basedeployment as a volume, so each entry is added as a file under the config directory:

                > cat base/deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: the-deployment
  annotations:
    app: nginx
spec:
  template:
    spec:
      containers:
      - name: nginxapp
        image: nginx:1.7.9
        env:
        - name: COMMON
          valueFrom:
            configMapKeyRef:
              name: config
              key: common
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
      volumes:
      - name: config-volume
        configMap:
          name: file-config
            

Now in the overlay, let’s add to our configMapGenerator:

                > cat overlays/prod/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
configMapGenerator:
- name: file-config
  behavior: merge
  files:
  - custom.properties
            

And now generating the config here is how it looks like:

                > k kustomize overlays/prod 
apiVersion: v1
data:
  common: Common Variable
kind: ConfigMap
metadata:
  name: config-mdd5k4b6c5
---
apiVersion: v1
data:
  common.properties: |-
    common=common_config
    other_common=common2_config
  custom.properties: |-
    custom1=custom1_config
    custom2=custom2_config
kind: ConfigMap
metadata:
  name: file-config-965d47772g
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    app: nginx
  name: the-deployment
spec:
  template:
    spec:
      containers:
      - env:
        - name: COMMON
          valueFrom:
            configMapKeyRef:
              key: common
              name: config-mdd5k4b6c5
        image: nginx:1.8.0
        name: nginxapp
        volumeMounts:
        - mountPath: /etc/config
          name: config-volume
      volumes:
      - configMap:
          name: file-config-965d47772g
        name: config-volume
            

Now the container can parse all the files specified under the /etc/config directory… all in all kustomize is pretty cool :) There are actually a lot more that it does, and this was just a quick overview of some of the functions.

I hope you found just how useful Kustomize can be to your daily needs, working with K8s. Please stay tuned and subscribe for more articles and study materials on DevOps, Agile, DevSecOps and App Development.

If you’d like to learn more about Infrastructure as Code, or other modern technology approaches, Please read or other articles.

#YouAreAwesome #StayAwesome


Only registered users can post comments. Please, login or signup.

Start blogging about your favorite technologies and get more readers

Join other developers and claim your FAUN account now!

Avatar

Apoti Eri

CEO, apotitech

@apotitech
A lover of Jesus and IT. I have a passion for teaching and sharing my knowledge with others. My very best time of the day is teaching and helping my peers.
Stats
20

Influence

1k

Total Hits

2

Posts