Scheduled backup of Vault secrets with CronJob of Kubernetes

#aws   #vault   #hashicorp   #Kubernetes  

The second part of the 'Scheduled backup of HashiCorp's Vault secrets', this time I will show how to use Kubernetes CronJob to create scheduled backups on daily basis.

I am a DevOps engineer at and I will share in this post my experience related to automation of Vault backup creation using Kubernetes CronJob.

This post is a continuation of the previous post:

The repository with all the code:

What is HashiCorp’s Vault?

A vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more. Vault provides a unified interface to any secret while providing tight access control and recording a detailed audit log.

My Setup

  • EKS Kubernetes cluster
  • Vault runs on EKS cluster

What you will learn from this post?

  • How to create a scheduled backup for Vault secrets with CronJob of Kubernetes.
  • How to add Prometheus alerts for failed jobs.

You can find all the code presented in my repository:

Let’s start.

Building the docker container

First, need to build a docker container based on python3 and include the code of

Need clone the repo first with Docker file: ‘git clone

Docker file:

FROM python:3
COPY requirements.txt /
RUN pip install -r requirements.txt
CMD [ "python", "./" ]

Building image

# login to dockerhub
$ docker login -u YOUR_USERNAME -p YOUR_PASSWORD
# Build Docker
$ docker build -t vault-backup .

Validate docker container working properly

$ docker run --name test-vault-backup --rm vault-backup
Specify one of the commands below

It’s working, we got a list of commands from vault-backup:-)

Pushing vault-backup docker container to docker hub

$ docker tag vault-backup <Your Docker ID>/vault-backup:latest
$ docker push <Your Docker ID>/vault-backup:latest

In my case, it’s ‘warolv/vault-backup:latest’, you can find an already built image there.

CronJob to run vault-backup on a daily basis

apiVersion: batch/v1beta1
kind: CronJob
  name: vault-backup
  schedule: "0 1 * * *"
          restartPolicy: Never
            instance-type: spot
            - name: awscli
              image: amazon/aws-cli:latest
                - "aws"
                - "s3"
                - "cp"
                - "/data/vault_secrets.enc"
                - "s3://jenkins-backups/vault_secrets.enc"
              imagePullPolicy: Always
                - secretRef:
                    name: aws-creds-secret
              - name: backup-dir
                mountPath: /data
            - name: vault-backup
              image: warolv/vault-backup
                - "python3"
                - ""
                - "dump"
                - "-dp"
                - "/data/vault_secrets.enc"
              imagePullPolicy: Always
                - secretRef:
                    name: vault-backup-secret
              - name: backup-dir
                mountPath: /data
          - name: backup-dir
            emptyDir: {}


  • First ‘’ script will run from InitContainer and secrets dump will be created (vault_secrets.enc) and saved to /data folder which is a shared folder for both containers.
  • The second will run ‘awscli’ container which will be used to push the secrets dump to a private S3 bucket (AWS CLI is used to copy the secrets dump to the privare S3 bucket). Of course, S3 private bucket must exist.
  • Credentials for AWS CLI (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY) and for vault_backup script exported to the environment as k8s secrets.
  • In this example, I am copying the dump to ‘s3://jenkins-backups/vault_secrets.enc’, in the production use case I suggest adding a timestamp to dump of secrets to be something like vault_secrets_${timestamp}.enc

Creating secrets for CronJob

# create k8s secret for AWS
$ kubectl create secret generic aws-creds-secret \

# create k8s secret with all needed data for vault-backup
$ kubectl create secret generic vault-backup-secret \
--from-literal=VAULT_ADDR=http://vault.vault.svc.cluster.local:8200 \
--from-literal=ROLE_ID=YOUR_ROLE_ID \
--from-literal=SECRET_ID=YOUR_SECRET_ID \

It’s only an example, you need to put real values.

Deploy vault-backup cronjob

$ kubectl apply -f examples/cronjob/cronjob.yaml

How to trigger a Job from CronJob?

In case you want to test your job is working properly:

$ kubectl create job --from=cronjob/vault-backup vault-backup-001

Adding alerts to Prometheus

I am using kube-state-metrics with Prometheus and we have these metrics available:

Let’s add an alert for ‘failed job’ and for cronjob which ‘takes too much time’, of course, it’s only an example to give you an idea.

- name: cronjob.rules
  - alert: SlowCronJob
    expr: time()-kube_cronjob_next_schedule_time > 1800
    for: 30m
      severity: warning
      description: CronJob {{$labels.namespaces}}/{{$labels.cronjob}} is taking more than 30m to complete
      summary: CronJob taking more than 30m
  - alert: FailedJob
    expr: kube_job_status_failed  > 0
    for: 30m
      severity: warning
      description: Job {{$labels.namespaces}}/{{$labels.job}} failed
      summary: Job failure


In this post, I described how to automate Vault backup creation using Kubernetes CronJob and a simple python script that I built.

Thank you for reading, I hope you enjoyed it, see you in the next post.

If you want to be notified when the next post of this tutorial is published, please follow me here on medium and on Twitter (@warolv).

Original story on my blog:

Start blogging about your favorite technologies and get more readers

Join other developers and claim your FAUN account now!


Igor Zhivilo

DevOps Lead, warolv

I am Software Engineer/DevOps who loves to learn and experiment with new technologies. // Kubernetes // Vault // Jenkins // Ruby on Rails // AWS // Cloud security // Currently work at



Total Hits

Discussed tools