How to prevent the IaC misconfiguration snowball effect

The goal with infrastructure as code (IaC) frameworks such as Terraform and CloudFormation is to make infrastructure provisioning more efficient. Through a combination of automation and either imperative or declarative configuration, IaC makes it easier to deploy the same environment consistently and repeatably.

IaC’s immutability and machine readability are huge advantages when it comes to building, deploying, and testing infrastructure. It allows for storing and versioning of infrastructure, making it easier to manage, collaborate, and audit. It also allows teams to test—and secure—infrastructure just as they would with any other code. But without the right approach, IaC can actually pose a disadvantage when it comes to security and compliance.

Misconfigurations-as-code

With IaC, all you need to deploy identical and purpose-built environments is defined and contained within your configuration files. This makes it incredibly easy to repeat and reuse at scale.

But what if that configuration is missing a security control or contains a security flaw?

When that happens, the value of using IaC to deploy at scale then becomes a nightmare when misconfigured IaC snowballs into tons of policy violations across multiple deployments or environments.

In the figure above, you can see how a single misconfiguration pre-deployment can propagate across hundreds of deployments, resulting in thousands of alerts for security teams. This so-called “snowball effect” happens when security tooling and infrastructure development practices don’t align and can negatively impact all parties involved.

Alert triage time

With manual or script-based cloud provisioning and runtime security monitoring in place, an unencrypted S3 bucket being provisioned should result in a few alerts. However, with logging and sufficient audit trails, it’s relatively easy to tie alerts to the exact resource in question and determine when, where, why, and how the misconfiguration was introduced.

With cloud resources provisioned with IaC, it’s not as straightforward. It’s more of a challenge (or requires additional infrastructure) to find the file and exact resource block that caused the issue. And with the snowball effect of IaC, the volume of alerts is much higher, which means infosec and DevOps have to spend considerably more time triaging issues. Those issues, in turn, snowball into tons of Jira or ServiceNow tickets that developers then need to address.

Productivity decrease

By the time a ticket reaches the right developer on the right team, the code in question is likely far behind them. And in that period, they’ve probably lost the full context of the resource or why the change was made in the first place. So tracing their steps and switching back into that context takes time.

Then, when the change is finally made, committed, and reviewed, it will need to go through your build pipeline to be deployed once again. All of this is time wasted that could be spent building new features and creating business value.

Shift left or drift trying

Addressing issues at their source and making the most of IaC’s immutability is the key to avoiding the snowball effect. Once a misconfiguration is identified, all you have to do is destroy and recreate it. That way, the old cloud resource is wiped out, a new, secure resource is put in its place, and your configuration files are always aligned to what’s actually deployed.

Unfortunately, this doesn’t always happen. Whether a hasty change gets made during a security incident or miscommunications result in out-of-band alterations, configuration drift is inevitable. Drift occurs when a resource in a running environment no longer matches its corresponding defined configurations. Although drift doesn’t always pose a security risk (often drift occurs when misconfigurations are fixed in situ by teams less familiar with the configuration that provisioned them), it should be avoided.

When drift occurs, your IaC code becomes less useful, and its benefits (versioning, auditability, repeatability) are no longer. Drift can also exacerbate the snowball effect of IaC. If misconfigurations aren’t addressed at the source, they will be resurfaced, and the cycle repeated every single time that IaC gets deployed.

To prevent the aforementioned snowball effect of IaC misconfigurations, security and development teams need to work together to embed security as early in the DevOps lifecycle. With the right tooling in place to periodically and continuously scan code—on developers’ local workstations, within shared repositories, and via CI pipelines—organizations can get visibility into misconfigurations before they’re deployed. By shifting security to the left, teams can cut back on runtime errors and configuration drift and improve time-to-fix while the context is still fresh.