Scheduled backup of Vault secrets with CronJob of Kubernetes

I am a DevOps engineer at Cloudify.co and I will share in this post my experience related to automation of Vault backup creation using Kubernetes CronJob.

This post is a continuation of the previous post: https://igorzhivilo.com/vault/scheduled-backup-vault-secrets/

The repository with all the code: https://github.com/warolv/vault-backup

What is HashiCorp’s Vault?

A vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more. Vault provides a unified interface to any secret while providing tight access control and recording a detailed audit log.

https://www.vaultproject.io/

My Setup

• EKS Kubernetes cluster
• Vault runs on EKS cluster

What you will learn from this post?

• How to create a scheduled backup for Vault secrets with CronJob of Kubernetes.
• How to add Prometheus alerts for failed jobs.

You can find all the code presented in my repository: https://github.com/warolv/vault-backup

Let’s start.

Building the docker container

First, need to build a docker container based on python3 and include the code of vault_handler.py.

Need clone the repo first with Docker file: ‘git clone https://github.com/warolv/vault-backup’

Docker file:

Building image

Validate docker container working properly

It’s working, we got a list of commands from vault-backup:-)

Pushing vault-backup docker container to docker hub

In my case, it’s ‘warolv/vault-backup:latest’, you can find an already built image there.

CronJob to run vault-backup on a daily basis

https://github.com/warolv/vault-backup/blob/main/examples/cronjob/cronjob.yaml

Explanation

• First ‘vault_backup.py’ script will run from InitContainer and secrets dump will be created (vault_secrets.enc) and saved to /data folder which is a shared folder for both containers.
• The second will run ‘awscli’ container which will be used to push the secrets dump to a private S3 bucket (AWS CLI is used to copy the secrets dump to the privare S3 bucket). Of course, S3 private bucket must exist.
• Credentials for AWS CLI (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY) and for vault_backup script exported to the environment as k8s secrets.
• In this example, I am copying the dump to ‘s3://jenkins-backups/vault_secrets.enc’, in the production use case I suggest adding a timestamp to dump of secrets to be something like vault_secrets_${timestamp}.enc

Creating secrets for CronJob

It’s only an example, you need to put real values.

Deploy vault-backup cronjob

How to trigger a Job from CronJob?

In case you want to test your job is working properly:

Adding alerts to Prometheus

I am using kube-state-metrics with Prometheus and we have these metrics available: https://github.com/kubernetes/kube-state-metrics/blob/master/docs/cronjob-metrics.md

Let’s add an alert for ‘failed job’ and for cronjob which ‘takes too much time’, of course, it’s only an example to give you an idea.

Conclusion

In this post, I described how to automate Vault backup creation using Kubernetes CronJob and a simple python script that I built.

Thank you for reading, I hope you enjoyed it, see you in the next post.

If you want to be notified when the next post of this tutorial is published, please follow me here on medium and on Twitter (@warolv).

Original story on my blog: https://igorzhivilo.com/vault/scheduled-backup-vault-cronjob/