Having used Kubernetes consistently at work, I had to find a tool that could make my work easier and much better. Let us get into Kustomize and see how to use it at our work.
Architecture
From their site, Kustomize has a very wide usage:
- Kustomize helps customizing config files in a template free way.
- Kustomize provides a number of handy methods like generators to make customization easier.
- Kustomize uses patches to introduce environment specific changes on an already existing standard config file without disturbing it.
It’s a perfect tool to create environment based customizations to your k8s deployments. kustomize uses a concept of bases and overlays, where you define a base and then you create overlays which customize the configuration depending on your environment. There is a pretty cool diagram in their github:
And here is a simple directory hierarchy:
~/someApp
├── base
│ ├── deployment.yaml
│ ├── kustomization.yaml
│ └── service.yaml
└── overlays
├── development
│ ├── cpu_count.yaml
│ ├── kustomization.yaml
│ └── replica_count.yaml
└── production
├── cpu_count.yaml
├── kustomization.yaml
└── replica_count.yaml
It’s also a perfect tool to be added into your CI/CD pipeline. From Workflow for off the shelf applications, here is a nice overview:
So let’s run through a couple of examples to see how it works. As a starting point, I created the following directory structure:
> mkdir -p {overlays/{dev,prod},base}
> touch {base/kustomization.yaml,overlays/{dev,prod}/kustomization.yaml}
> tree
.
├── base
│ └── kustomization.yaml
└── overlays
├── dev
│ └── kustomization.yaml
└── prod
└── kustomization.yaml
4 directories, 3 files
Container Images
First let’s try out modifying the container image version (this example is covered in customizing). As an example let’s create a simple deployment manifest:
> cat base/deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: the-deployment
annotations:
app: nginx
spec:
template:
spec:
containers:
- name: nginxapp
image: nginx:1.7.9
And here is a simple kustomization to modify the tag:
> cat base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yaml
images:
- name: nginx
Now to see how it gets parsed, we can use kubectl kustomize:
> k kustomize base
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
app: nginx
name: the-deployment
spec:
template:
spec:
containers:
- image: nginx:1.8.0
name: nginxapp
To actually apply it, we can run kubectl apply -k (vs kubectl apply -f):
k apply -k base
deployment.apps/the-deployment created
That was pretty easy.
Patching
kustomize has two approaches to patching files: strategic merge patch and json 6902 patch.
Strategic Merge Patch
This is useful if you are making a lot of changes, so you can just provide a YAML file that looks like a k8s manifest file with all the resources you want to add/change. From the Patching multiple resources at once example, let’s inject a side car container into our deployment in our dev overlay. First let’s add a kustomization.yaml file into the dev overlay:
> cat overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: deploy-patch.yaml
And now let’s add the patch file to specify the container we want to add:
> cat overlays/dev/deploy-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: the-deployment
spec:
template:
spec:
containers:
- name: istio-proxy
image: docker.io/istio/proxyv2
args:
- proxy
- sidecar
And here is how the parsed manifest will end up looking like:
> k kustomize overlays/dev
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
app: nginx
name: the-deployment
spec:
template:
spec:
containers:
- args:
- proxy
- sidecar
image: docker.io/istio/proxyv2
name: istio-proxy
- image: nginx:1.8.0
name: nginxapp
That looks good.
JSON 6902 Patch
This is good for adding/modifying one field at a time (also the parent path has to exist, so it doesn’t create sub fields, for more information check out this github feature request). Let’s say I want add a new annotation and to a serviceaccount, we could create the following patch:
> cat overlays/dev/deploy-more-patch.yaml
- op: add
path: /metadata/annotations/app.io~1owner
value: "me"
- op: add
path: /spec/template/spec/serviceAccountName
value: app
Notice that I had to use ~1 to specify a forward slash / (this is covered in this github issue). And here is update kustomization.yaml to include the new patch file. Notice that I also added a target to specify which resource to modify:
> cat overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: deploy-patch.yaml
- path: deploy-more-patch.yaml
target:
kind: Deployment
And now here is the resulted manifest:
> k kustomize overlays/dev
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
app: nginx
app.io/owner: me
name: the-deployment
spec:
template:
spec:
containers:
- args:
- proxy
- sidecar
image: docker.io/istio/proxyv2
name: istio-proxy
- image: nginx:1.8.0
name: nginxapp
serviceAccountName: app
yay, that worked out as well.
ConfigGenerator
Let’s cover one more example, the configMapGenerator, this supports multiple approaches as well. I will cover two.
Using Literals to Specify Configs as Strings
There is also a nice example at ConfigMap generation and rolling updates. Let’s say I create the following in my base:
> cat base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yaml
images:
- name: nginx
newTag: 1.8.0
configMapGenerator:
- name: config
literals:
- common="Common Variable"
Now at the overlay kustomization:
> cat overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: deploy-patch.yaml
- path: deploy-more-patch.yaml
target:
kind: Deployment
configMapGenerator:
- name: config
behavior: merge
literals:
- custom="My Custom Config"
Now I can modify the the deployment in the base to add the common configMap:
> cat base/deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: the-deployment
annotations:
app: nginx
spec:
template:
spec:
containers:
- name: nginxapp
image: nginx:1.7.9
env:
- name: COMMON
valueFrom:
configMapKeyRef:
name: config
key: common
And then I added a patch in the overlay to append the environment variable (since I my sidecar proxy patch from before added up adding the container first, I had to basically append it to the second container of my deployment… 0 is the first and 1 is the second):
> cat overlays/dev/deploy-more-patch.yaml
- op: add
path: /metadata/annotations/app.io~1owner
value: "me"
- op: add
path: /spec/template/spec/serviceAccountName
value: app
- op: add
path: /spec/template/spec/containers/1/env/-
value:
name: CUSTOM
valueFrom:
configMapKeyRef:
name: config
key: custom
Parsing the config, here is what it creates:
> k kustomize overlays/dev
apiVersion: v1
data:
common: Common Variable
custom: My Custom Config
kind: ConfigMap
metadata:
name: config-k96cb96f75
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
app: nginx
app.io/owner: me
name: the-deployment
spec:
template:
spec:
containers:
- args:
- proxy
- sidecar
image: docker.io/istio/proxyv2
name: istio-proxy
- env:
- name: COMMON
valueFrom:
configMapKeyRef:
key: common
name: config-k96cb96f75
- name: CUSTOM
valueFrom:
configMapKeyRef:
key: custom
name: config-k96cb96f75
image: nginx:1.8.0
name: nginxapp
serviceAccountName: app
We can see it creates a dynamic configMap name and adds it to our container accordingly… that is pretty cool :)
Using Files to Specify Configs
Another approach is to provide configuration files in the configMaps. Each file ends up being a new data entry in the configMap. Another cool example at Demo: combining config data from devops and developers. Let’s add it to our prod overlay. First let’s modify our base and add a new configMapGenerator:
> cat base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yaml
images:
- name: nginx
newTag: 1.8.0
configMapGenerator:
- name: config
literals:
- common="Common Variable"
- name: file-config
files:
- common.properties
Now let’s add it to our basedeployment as a volume, so each entry is added as a file under the config directory:
> cat base/deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: the-deployment
annotations:
app: nginx
spec:
template:
spec:
containers:
- name: nginxapp
image: nginx:1.7.9
env:
- name: COMMON
valueFrom:
configMapKeyRef:
name: config
key: common
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
name: file-config
Now in the overlay, let’s add to our configMapGenerator:
> cat overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
configMapGenerator:
- name: file-config
behavior: merge
files:
- custom.properties
And now generating the config here is how it looks like:
> k kustomize overlays/prod
apiVersion: v1
data:
common: Common Variable
kind: ConfigMap
metadata:
name: config-mdd5k4b6c5
---
apiVersion: v1
data:
common.properties: |-
common=common_config
other_common=common2_config
custom.properties: |-
custom1=custom1_config
custom2=custom2_config
kind: ConfigMap
metadata:
name: file-config-965d47772g
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
app: nginx
name: the-deployment
spec:
template:
spec:
containers:
- env:
- name: COMMON
valueFrom:
configMapKeyRef:
key: common
name: config-mdd5k4b6c5
image: nginx:1.8.0
name: nginxapp
volumeMounts:
- mountPath: /etc/config
name: config-volume
volumes:
- configMap:
name: file-config-965d47772g
name: config-volume
Now the container can parse all the files specified under the /etc/config directory… all in all kustomize is pretty cool :) There are actually a lot more that it does, and this was just a quick overview of some of the functions.
I hope you found just how useful Kustomize can be to your daily needs, working with K8s. Please stay tuned and subscribe for more articles and study materials on DevOps, Agile, DevSecOps and App Development.
If you’d like to learn more about Infrastructure as Code, or other modern technology approaches, Please read or other articles.
#YouAreAwesome #StayAwesome
Share with your friends and followers
Start blogging about your favorite technologies, reach more readers and earn rewards!
Join other developers and claim your FAUN account now!
Apoti Eri
CEO, apotitech
@apotitech
User Popularity
62
Influence
5k
Total Hits
2
Posts
Read, Learn, Know, Teach
Hand curated newsletters for Developers, private Slack with like minded people, podcasts, job offers, news and more!
Only registered users can post comments. Please, login or signup.