How We Improved Our Monitoring Stack With Only a Few Small Changes

The article discusses the process of improving the monitoring system at Riskified.

The team identified pain points and goals, including bottleneck on changes in the monorepo, crashing Prometheus, inability to silence alerts easily, and removing hardcoded secrets in Alertmanager config.

They considered various options to improve the system, including a single Alertmanager, Thanos ruler , and Prometheus-operator. Ultimately, they chose to use Prometheus-operator to split the Alertmanager configuration for each team and use a native Alertmanager configuration file stored in a Kubernetes secret.

They also used Pager Duty to manage alert silences. The team learned that small changes can make a big difference and encourages others to share their experiences to improve and optimize their own monitoring systems.