Unit Testing Alerting with the Prometheus Operator

At loveholidays, Prometheus and Alertmanager are used for alerting based on metrics provided by applications and the Linkerd service mesh. They employ Prometheus operator to deploy and scale a company-wide Prometheus service, which allows defining alerts using PromQL queries and thresholds. To ensure alerts fire as expected, unit testing for Prometheus alerts is crucial, and this can be achieved using Prometheus unit testing, as described in the article. Additionally, a tool called "po-test" has been open-sourced to simplify testing PrometheusRule manifests alongside unit tests for Prometheus, making it more convenient for both local and CI use.