The article discusses the process of improving the monitoring system at Riskified.
The team identified pain points and goals, including bottleneck on changes in the monorepo, crashing Prometheus, inability to silence alerts easily, and removing hardcoded secrets in Alertmanager config.
They considered various options to improve the system, including a single Alertmanager, Thanos ruler , and Prometheus-operator. Ultimately, they chose to use Prometheus-operator to split the Alertmanager configuration for each team and use a native Alertmanager configuration file stored in a Kubernetes secret.
They also used Pager Duty to manage alert silences. The team learned that small changes can make a big difference and encourages others to share their experiences to improve and optimize their own monitoring systems.
Share with your friends and followers
Start blogging about your favorite technologies, reach more readers and earn rewards!
Join other developers and claim your FAUN account now!
Only registered users can post comments. Please, login or signup.