Unleash the full power of automation with self-service Internal Developer Portals

Discover in this story how to boost the productivity of your engineers with an Internal Developer Platform and what are the key features to make it successful.

Automation, a sad story…

Once upon a time 🦄, when your company was small enough so that a handful of people was building and running all the software and the infrastructure, the operations were very smooth and it was very easy to ship something to production, your teammate (most probably the cofounder) would sit right next to you and deploy your application with you, or they would even give you full access to do it yourself (security was not yet a concern at that time).

Then, as the company grows, more and more people are starting to come in and bring value by building more performant but also more complex systems. Now, in order to deploy your application you need a skilled team of engineers who knows how the whole system works and that know how to keep it running and in good shape, a big applause to the infra team!

But the company keeps growing and growing and your infra team needs to find an ally in order to keep up with the always-rising number of requests. Here is where Automation starts to kick in.

Lots of new tools are introduced to ease the day-to-day tasks of the infra team, and they can now deploy things with a couple of commands using Infrastructure as Code.

Sure, automation is already a good start, but we wouldn’t unlock its full potential if we still need to rely on someone to know how and where to kick in that automation. Let me explain why.

Most of the time infrastructure automation is done thanks to CD pipelines, which given an input in a specific format from a GitHub Pull Request will perform some actions in order to deploy things with the right configurations. The issue with pipelines is that you need to know “where” and “how” to trigger them, which in an always-growing environment can become challenging. You’ve probably seen people organizing meetings to do pull requests together with the DevOps/SRE, or frustrated developers who can’t get approval for their PR as it’s not in the right format or syntax is wrong.

All this has to end!

It should be easier for people to perform those operations in full autonomy and they shouldn’t spend their valuable time searching and learning the specific YAML format needed to deploy things.

This is where Internal Developer Portals come to rescue us.

What is an Internal Developer Portal (IDP)?

A Developer Portal is a go-to place for developers when they need to learn about their ecosystem and contribute to it. Developer Portals exist to balance Developer Autonomy and Discoverability: it allows teams to keep working independently, choosing their preferred tools, and deploying frequently, but expose their work so everyone can explore and benefit from their microservices, APIs, libraries, websites, pipelines, and algorithms.

What are the main features to look into when choosing an IDP solution?

There are many alternatives of IDPs on the market: open-source, paid, and free. My objective here is not to give you a benchmark on which is the best but to give you the tools to choose the most successful for your organization according to what I believe are the most important features.

A successful IDP needs to serve three purposes:

• helps developers navigate their ecosystem
• empowers engineers with self-service capabilities that follow the golden path defined by the company
• provides leaders and teams with insight into the tech’s health and maturity

Navigate your ecosystem with the Catalog

The catalog is a central repository that contains all of the APIs, SDKs, and other resources that are available to developers within an organization. The catalog typically provides developers with a comprehensive overview of the available resources, including documentation, sample code, and other relevant information. It may also include features such as search functionality, filtering, and sorting to help developers quickly find the resources they need.

What is important to look at in this component is how easily you can get meaningful data into the catalog. This includes both how to better represent the entities and their relations and from which sources you can easily pull information from.

There are many alternatives in the market but we can summarize them in two types:

• Pull based
• Push based

The first relies on making direct calls to the components ( cloud providers, Kubernetes API, … ) or to some GitOps repo in order to pull the data, while the latter leverage API to push new data from various sources.

My 2 cents here, Push (API-based) approach is more flexible, allowing you to push data from CI pipelines or custom exporters that you can easily write in your preferred language.

Self-Service capabilities with Scaffolder

Scaffolder is a tool that helps developers to quickly create new applications or projects by generating a starting point for their code. This can include a basic application structure, pre-configured settings and sample code that developers can build upon. Scaffolder may also provide guidance and best practices for developers to follow, helping to ensure that applications are built in a consistent and scalable manner.

Most of the IDPs allow you to make use of GitOps patterns in order to provision a new repository or a new resource in an already existing environment thanks to underlying IaC. Most of them make use of Cookiecutter or a similar approach.

Self-Service capabilities with Day-2 Operations

This is a capability that not all the IDP offers, but that I think is fundamental to really empower developers to run their own infrastructure without granting them access to production directly.

Day-2 Operations can go from “Restart a Service” to “Deploy a temporary test environment” or to “Create a new DB”.

The important thing to look at in this feature is how easily you can hook your preferred automation to the actions in the IDP. For example, you might want to leverage GH action to perform the automation action, Terraform, or whatever tool is in your day-to-day operations.

Gather insights into your ecosystem maturity and respect the standards

Once your ecosystem of services starts to flourish it will be difficult to keep track of what is the production readiness or the security posture of a service. Here is where the Catalog, thanks to the data it gathers can give you visibility over these important KPIs.

Most of the IDPs allow you to either pull data from external sources or use the information on the Catalog and evaluate them as part of what they call “Scorecards”. Scorecards allow you to define what are the different levels of maturity you expect from a service by linking it to levels like “Bronze”, “Silver” and ”Gold”. In this way, it becomes very easy to focus your effort on getting everyone on the same Golden Path.

Final Thoughts

There are many choices to make around automation and Internal Developer Portal. Most of the time you’ll be able to find and follow the patterns and best practices of people that already took the path leading to this amazing journey. But many times you’ll be also wandering in uncharted territory and on top of that you’ll need to be able to make this new concept, work well with the preexisting infrastructure automation and processes of your organization.

So I wish you good luck on this journey, I hope you find this article insightful and please feel free to share your experiences in the comments section.

Ah and make sure to follow me on Medium for more articles!😃 ->

Alessio Trivisonno

Credits:

image: Designed by pch.vector / Freepik