They Stopped Using Load Tests - And Here Is Why

Many companies dropped load tests. Even though for many years we relied on load tests as the only way to prevent scalability issues, we now have other ways. Let’s see what happened and why you should drop load tests too.

Why Load Tests Do Not Work

Load tests are designed to determine whether the service and database are sufficiently fast for production deployment. While they are effective in this role, they come with numerous drawbacks that suggest they should be avoided.

First, load tests are time-consuming. To obtain meaningful results, we must execute thousands of transactions over several hours, ensuring caches, drives, and networks are sufficiently filled and active for an extended period. This means we cannot obtain early results, typically letting tests run overnight or triggering them in the CI/CD pipeline without waiting for immediate feedback.

Second, load tests are challenging to create. Simply bombarding the service with random requests isn't sufficient; we must replicate production data distribution and send logically consistent data. This involves managing stateful services, preparing databases, and ensuring comprehensive coverage of all code paths. This task becomes particularly complex when the service behaves differently across various regions or evolves rapidly over time, making it difficult to achieve meaningful results.

Third, maintaining load tests is difficult. As data and service implementations change over time, we must constantly update our testing data. While replaying production traffic might seem like a solution, it risks creating invalid states and missing critical code paths.

Fourth, compliance with regulations such as GDPR and CCPA adds complexity. We cannot simply use production data in non-production environments due to security policies and the risk of data breaches. This necessitates anonymizing data, removing sensitive information, and ensuring customer data remains secure, which is both challenging and risky.

Fifth, randomness complicates the process. It's normal to receive different responses for the same request due to variations in timestamps, random identifiers, and other factors. This makes writing assertions more difficult, as we cannot compare results byte by byte and must instead understand and compare their semantics.

Sixth, load tests often occur too late in the development process. Discovering issues during load testing means the code has already been written, reviewed, merged, and deployed to some environments. Fixing these issues at this stage is costly and requires significant effort, often necessitating a complete reevaluation of the approach.

Simply put, load tests have many drawbacks. They are slow, challenging to write and maintain, and very expensive to run over time. Can we do better?

What to Do Instead?

Instead of relying on load tests, we should aim to catch issues early in the development process. Various observability techniques can help us monitor behind-the-scenes activities and identify problems in the developers' environments.

By using telemetry, we can capture the queries sent to the database and project them onto the production database to extract execution plans. This allows us to immediately determine if the queries will be fast enough for production deployment.

We can apply the same approach to verify schemas and configurations. By checking if indexes are properly configured and if queries utilize them efficiently, we can ensure optimal data retrieval. This process can be automated and compared with the production database, providing developers with immediate feedback within their environments, and eliminating the need for code reviews or staging deployments.

Observability offers even more capabilities. Understanding what changes are made and how they affect the system allows us to pinpoint issues in production and correlate them with specific code changes. This enables us to submit automated pull requests to fix these issues, and tune configurations, schemas, indexes, and extensions. Ultimately, this approach ensures database reliability and facilitates automated self-healing.

Should We Drop Load Tests?

Taking all of that into consideration, should we drop the load tests entirely? Yes, we should. Our ultimate goal should be to have no load tests at all. We can achieve reliability without running long and expensive testing procedures. Observability and telemetry give us reliability and predictability without the hassle of running load tests. Instead of writing the tests, focus on improving your observability.