Best Practices for Testing Zone Redundancy

Amazon's early practice of cutting power to test redundancy might be apocryphal, but today's engineers confront a variety of zonal failures such as packet loss or bad configurations, with best practices evolving to include methods like over-provisioning and load testing to ensure resilience—highlighting that merely distributing services across multiple zones doesn't guarantee resilience; instead, comprehensive testing, like simulating zonal failures, is crucial to validate if systems can truly withstand regional outages, taking into account potential pitfalls like cross-zone dependencies and autoscaling issues.