Continuous Integration — CI & Continuous Delivery —CD | How to build a high-quality pipeline

It´s worth investing a little time to understand continuous integration (CI) and continuous delivery (CD), mainly in the development software world in which exists are builds and controls version repositories.

CI — Continuos Integration

CI is a process that allows us to make changes quickly and easily without being painful.
Newman, Sam. Building Microservices . O’Reilly Media. Kindle Edition.

The main goal of CI is to keep the team in sync with the existing code from the repository. Basically, a good CI detects that the code has been committed, and runs some verifications such as if the code is compiling and if the tests are passed.

This process brings us many benefits, such as fast feedback about the application. This process allows us in each modification to have faster feedback about the last changes made, if any flow was broken, for example, we can do this over and over again, gaining more confidence when we need to update a feature, o refactor any piece of the code.

Tools it’s not enough

Sometimes we think that because we have a Jenkins or Circle CI or any tools that you know, have been installed and working in the CI server, we believe… ok I use CI in my team, but in fact, to have these tools it’s not enough.

In my career, I’ve had the opportunity to see what these big guys, Sam Newman and Martin Fowler explain about this process.

CI the right way

Should have a suite of tests — In a team is possible to know integrations of that new feature a has been worked but isn’t possible to know if that new feature has broken the behavior of the system. Also, a suite of tests helps not promote a refactor fear.
Do integration every day — You need to integrate your code with mainline every day, if you don’t do this, you will be making future integrations harder, even if you use short branches, integrate as frequently, at least once per day.
When the build is broken the first priority of the team is to fix it — In my career I saw builds broken for weeks, causing stress for new deliveries, so to avoid this, the priority of the team is to fix it as quickly as possible.

Building high-quality pipeline

I’ll share with you some format pipelines I saw in my career.

In a pipeline, there are stages that run some tasks that need to validate and build the application which will be deployed.

Dream pipeline img-1.0

In img-1.0 above, is a great form to build a pipeline process, basically, the code source is a git serve, for example, the CI serve does checkout this code source. After checkout, the pipeline can build the artifact by source code.

The pipeline has some stages that all CI processes should have

Unit Test — After checkout from the code, the first thing is to run all unit tests from the application and if any test is broken, the pipeline is interrupted
Integration test — After running the unit test normally is run all integration tests from the application, and if any test is broken, the pipeline is interrupted. Sometimes I saw these steps running in parallel, but I think not to make sense because the integration test is a step that slower than the unit test step.
E2E test — ok if in your application there is an E2E test is a good practice to put it after the integration test
Migration — Here it’s a step that normally many teams don't put in your CI process, for many reasons, some are: another team is the owner of the database or security reasons, they prefer to run a migration script on your laptop, or there is another pipeline for executing the migration of the database.
Build — After the steps before running with success, it's time to build the artifact and deploy the new version of the application on any server.
Publish — This step basically sends the artifact of the application to the server responsible for receiving requests

Why don’t run all tests together?

The main reason is unit tests are faster than integration tests, so if any unit test has broken, you get feedback faster by running the unit tests first.

Continuous Integration & Continuous Delivery by environments

There are ways to build a CI process for different environments, it is necessary because, for example: if a software engineer is developing a new feature yet, he can run the pipeline and deploy this new feature until the staging environment for executing exploratory tests.

Below there are other pipeline examples

Pipeline without migration stage IMG-1.1

In IMG-1.1 above is a pipeline without a migration stage, if the software engineer needs to do structural modification, he needs to run a migration script in your workstation or in another pipeline.

Step by step from the Engineer running a migration script for each environment

There are many migration tools such as Liquibase and Flyway

CI servers by environment — IMG-1.2

In IMG-1.2 above, are pipelines for each environment sometimes some teams have the preference split the CI server by environment instead keep a unique pipeline, the reason is security, and these teams work with Feature Branch normally, for example, to trigger the production pipeline should have a commit for the main branch.

Another way of working with a pipeline is adding manual releasing for some environments

CI servers by environment and filtered by branch— IMG-1.3

In IMG-1.3 it is very common for many teams to use this way, to prepare a version of code for production but only release it manually.

Conclusion

CI process it’s not just having a server with CI tools, but you made and keep a process in a team.

The CI process improves the confidence of software and provides faster feedback and reduces bugs in production, the CI tools run these processes.

Continuous Delivery (CD) — means that if the process of tests, and builds are being run with every commit and validated, then it is possible to release the software to production at any time, as we see in examples such as IMG-1.3 each commit to your code is ready for production but does not being release to production.

References

Building Microservices : Designing Fine-Grained Systems