Observability with Prometheus and Grafana

What is Network Black Box Monitoring?

The term "black box" is widely used in various fields and industries. For example, it can be used to describe a method of software testing where the functions of an application are tested without examining its internal structure or workings. In engineering, a black box is a device or system whose internal workings are not known or accessible to the user. In aviation, a black box refers to the flight data recorder used to investigate accidents. In the context of monitoring, black box monitoring refers to checking system metrics such as disk, CPU, and memory usage without focusing on the details of the system or the applications deployed on it. White box monitoring, on the other hand, involves monitoring applications and systems from the inside. In all these contexts, the common theme is that the internal details are hidden or unknown, and only the inputs and outputs are considered.

Black box monitoring applied to network monitoring means monitoring the network from the perspective of an end user or a client, not from inside the network. This approach is useful for understanding the overall user experience, the abstracted view of the network, and the services it provides. There is another case where we are forced to use black box monitoring: when it's not possible to instrument the service with Prometheus metrics.

For example, you may want to monitor the expiration date of an SSL certificate that you bought from a third-party provider. In this case, you can't install a Prometheus exporter on the provider's infrastructure, but you can still monitor the expiration date of the certificate from the outside using black box monitoring.

Imagine you are using managed services like a load balancer, an ingress server, a DNS server, or a database service provided by a cloud provider. In this case, you don't have access to the internal metrics of these services. In this case, you are helpless when it comes to white box monitoring; nevertheless, black box monitoring comes to the rescue.

ℹ️ There's a confusion - or probably a hype - around white box monitoring being better than black box monitoring since it's more related to observability. This is a misconception. Comparing these two approaches should not be about which one is better, but about which one is more suitable for a specific use case. You may need to use both approaches in your monitoring/observability as they serve complementary purposes.