Building a resilient system: Our journey to observability at Intercom

Intercom prioritizes customer experience, and to achieve that, they invest heavily in the reliability of their application.

They operate a socio-technical system, and the ability to recover from adversity is called resilience, and observability is a crucial component of resilience.

Observability is defined as a continuous process of humans asking questions about production and getting answers. Previously, Intercom relied on metrics for observability, but it had limitations in investigating unpredictable failures. Therefore, Intercom decided to invest in tracing telemetry to enable a smooth consolidated experience without the need to switch the underlying data or the tool.

After assessing their existing practices and formulating a problem statement, they shifted towards traces and chose Honeycomb as their vendor. They also focused on building a culture of observability, enabling teams to adopt new observability tooling, and measuring adoption.