Deploy a GCloud instance with Gitlab CI

Today we’re going to discuss a tool that I love using every day: Gitlab. It has grown into more than just a Git repository but a veritable arsenal of DevOps technologies in a few years. We will focus more on its CI / CD pipeline, but we will use other features like its Docker image registry or its Terraform state backend. The project is quite simple: deploy and provision a GCE Instance with Terraform and Ansible automatically from a simple Git commit.

Project architecture

The project file will be structured as follows:

• An ansible folder with a playbook named main.yml. You can also add roles if you want
• A terraform folder containing all the Terraform files needed to create our virtual machine: main.tf, outputs.tf, backend.tf, variables.tf and terraform.tfvars
• A .gitlab-ci.yml file defining all the stages of our CI / CD pipeline to reach our end
• A Dockerfile allowing to build an Ansible custom image
• A .gitignore file to not commit the Terraform state files as well as the terraform.tfvars file

Configuring Terraform and Ansible

The main.tf file contains a fairly basic configuration for deploying a VM on Google Cloud. Many values ​​are variable to allow flexibility when running the pipeline:

Line 22: We install python3 on the machine at startup so that Ansible can perform its actions without problems

Line 24–26: We import a public key so that Ansible can connect without a password to the virtual machine. Note that the value of the gcp_user variable must be your Google account username.

The outputs.tf file must define an output variable that will be the machine’s public IP. We will use this variable to tell Ansible on which host it should run its playbook.

The backend.tf file creates an HTTP backend for Terraform.

For Ansible, it’s even easier! I’m just creating a playbook that will display variables called ansible_facts that Ansible will retrieve during its harvest phase by connecting to the host.

Creating the Docker Ansible image

The CI / CD pipelines on gitlab.com public runners use Docker containers to run whatever commands you want, so you need a Terraform image and an Ansible image. For the first, Gitlab provides one, but, surprisingly enough, there is no official Docker image for Ansible. So we will need to create it to run our playbook.

I decided to go with an archlinux as a base image because my computer is running Manjaro, and I don’t want compatibility issues with the Ansible versions. But, it is quite possible to start on another basis; there are plenty of examples on the Internet.

Either way, the Dockerfile doesn’t contain anything foolish: I install the packages needed for Ansible, the jumit_xml library (you’ll see why later), I turn off host checking for SSH, and I create a generic user who will run the playbook. I could have added collections or Ansible modules as well as dependencies in python if I needed to.

We’ll automatically build this image in our pipeline, upload it into the built-in registry on Gitlab, and then use it in later steps.

Gitlab CI pipeline

Here we get to the heart of the matter! I took as an example the model that Gitlab provided for a Terraform pipeline available here. From this example, I removed the job to delete the ‘terraform destroy’ command and put the ‘build’ job in automatic mode instead of manual.

I defined the stages of my pipeline like this:

• Preparation: The preparation step for the build will contain the jobs initializing the Terraform workspace and building the Docker Ansible image.
• Validation: Validation step performing syntax tests on Terraform and Ansible files.
• Building: Step performing a “terraform plan” to create the infrastructure.
• Deploying: Stage building the infrastructure and retrieving the machine’s IP address.
• Provisioning: Stage running the Ansible playbook.

I used Gitlab’s “needs” keyword to tell it which tasks are dependent on others. For example, this makes it possible to launch the job’ plan’ after the job ‘validation’ without waiting for the ‘lint’ job (the lint of Ansible). With this rearrangement, the pipeline can run much faster.

Also, for the same optimization purpose, I told Gitlab to run the ‘docker-build’ job only if I performed modifications on the Dockerfile.

In this pipeline, I also used the new features of Gitlab, which can now record the state of the Terraform infrastructure to have consistent builds. To do this, you only need to add a TF_ADDRESS variable indicating the address of the Terraform backend containing the state of the infrastructure:

To allow passing the variable containing the IP address of the virtual machine created in the “apply” job to the job running the Ansible playbook, we must use a “dotenv” report in the “artifacts” part of our job:

All jobs executed after this one can use the ANSIBLE_HOST environment variable.

And finally, I used the functionality of Gitlab, which allows creating JUnit test reports in its interface. I used the JUnit module to export the Ansible logs in the correct format, hence installing the junit_xml library. Next, you must define the environment variables ANSIBLE_STDOUT_CALLBACK and JUNIT_OUTPUT_DIR before calling the playbook. In the “artifacts” part, we specify the path of the results files in XML format.

You can see the final content of the .gitlab-ci.yml file in this gist

Let’s deploy!

First of all, we will generate an SSH key pair so that Ansible can connect to the GCloud VM with the following command:

In addition, on Gitlab, we must enter the environment variables of our CI / CD pipeline. I used the TF_VAR_ {var} syntax to declare the Terraform variables; it will then take them into account at runtime. Remember to specify the File type for the SSH keys and the GCloud credentials file:

We can now push our code and see the pipeline run automatically! On our GCloud console, we can see the virtual machine created with the name of the project. You can navigate several menus on Gitlab to see the performed actions.

Terraform menu in the Infrastructure tab

We can download the state of our Terraform infrastructure or even block it to prevent it from being modified.

JUnit tests in our pipeline

We can consult all our Ansible playbook’s actions, their status, and detailed logs.