Automating Managed Prometheus and Grafana with Terraform for scalable observability on Azue

The author shares their experience of using Azure Managed Grafana and Azure Monitor Managed service for Prometheus with Istio to improve observability in a Kubernetes cluster. The Terraform code provided automates the deployment of Azure Kubernetes Service, Azure Monitor Managed service for Prometheus, Azure Managed Grafana, and Istio.

• Using managed observability offerings, like Azure Managed Grafana and Azure Monitor Managed service for Prometheus, can save time and effort for small platform teams, allowing engineers to focus on the product rather than the observability platform.
• The Terraform code provided in the repository can automate the deployment of Azure Kubernetes Service, Azure Monitor Managed service for Prometheus, Azure Managed Grafana, and Istio.
• When using Azure Managed Grafana, it is important to correctly assign role assignments.
• When using Istio, it can be helpful to scrape the Istio sidecar Prometheus endpoint to obtain networking metrics.
• To improve observability and enforce mTLS for encryption in transit for traffic between workloads, Istio can be configured to scrape using Istio certificates.
• When using Azure Monitor Managed service for Prometheus, the Istio control plane, gateway, and Envoy sidecar metrics will be scraped over plaintext. To have the scraping continue to work, a specific PeerAuthentication with a portLevelMtls field can be written to disable the scraping port.