The second part of the 'Scheduled backup of HashiCorp's Vault secrets', this time I will show how to use Kubernetes CronJob to create scheduled backups on daily basis.
I am a DevOps engineer at Cloudify.co and I will share in this post my experience related to automation of Vault backup creation using Kubernetes CronJob.
This post is a continuation of the previous post: https://igorzhivilo.com/vault/scheduled-backup-vault-secrets/
The repository with all the code: https://github.com/warolv/vault-backup
What is HashiCorp’s Vault?
A vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more. Vault provides a unified interface to any secret while providing tight access control and recording a detailed audit log.
My Setup
- EKS Kubernetes cluster
- Vault runs on EKS cluster
What you will learn from this post?
- How to create a scheduled backup for Vault secrets with CronJob of Kubernetes.
- How to add Prometheus alerts for failed jobs.
You can find all the code presented in my repository: https://github.com/warolv/vault-backup
Let’s start.
Building the docker container
First, need to build a docker container based on python3 and include the code of vault_handler.py.
Need clone the repo first with Docker file: ‘git clone https://github.com/warolv/vault-backup’
Docker file:
Building image
Validate docker container working properly
It’s working, we got a list of commands from vault-backup:-)
Pushing vault-backup docker container to docker hub
In my case, it’s ‘warolv/vault-backup:latest’, you can find an already built image there.
CronJob to run vault-backup on a daily basis
https://github.com/warolv/vault-backup/blob/main/examples/cronjob/cronjob.yaml
Explanation
- First ‘vault_backup.py’ script will run from InitContainer and secrets dump will be created (vault_secrets.enc) and saved to /data folder which is a shared folder for both containers.
- The second will run ‘awscli’ container which will be used to push the secrets dump to a private S3 bucket (AWS CLI is used to copy the secrets dump to the privare S3 bucket). Of course, S3 private bucket must exist.
- Credentials for AWS CLI (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY) and for vault_backup script exported to the environment as k8s secrets.
- In this example, I am copying the dump to ‘s3://jenkins-backups/vault_secrets.enc’, in the production use case I suggest adding a timestamp to dump of secrets to be something like vault_secrets_${timestamp}.enc
Creating secrets for CronJob
It’s only an example, you need to put real values.
Deploy vault-backup cronjob
How to trigger a Job from CronJob?
In case you want to test your job is working properly:
Adding alerts to Prometheus
I am using kube-state-metrics with Prometheus and we have these metrics available: https://github.com/kubernetes/kube-state-metrics/blob/master/docs/cronjob-metrics.md
Let’s add an alert for ‘failed job’ and for cronjob which ‘takes too much time’, of course, it’s only an example to give you an idea.
Conclusion
In this post, I described how to automate Vault backup creation using Kubernetes CronJob and a simple python script that I built.
Thank you for reading, I hope you enjoyed it, see you in the next post.
If you want to be notified when the next post of this tutorial is published, please follow me here on medium and on Twitter (@warolv).
Original story on my blog: https://igorzhivilo.com/vault/scheduled-backup-vault-cronjob/
Share with your friends and followers
Start blogging about your favorite technologies, reach more readers and earn rewards!
Join other developers and claim your FAUN account now!
Igor Zhivilo
DevOps Lead, warolv
@warolv
User Popularity
91
Influence
9k
Total Hits
2
Posts
Read, Learn, Know, Teach
Hand curated newsletters for Developers, private Slack with like minded people, podcasts, job offers, news and more!
Only registered users can post comments. Please, login or signup.