How Squadcast Benefits On-call Engineers - Part 1

“Being on-call sucks!"

Often incident response teams use this phrase when talking about their on-call experiences. Despite using best practices for managing infrastructure, incidents do occur from time to time.

In order to avoid delays in responding to incidents and prevent being overwhelmed by on-call notifications, you should find a solution that helps in resolving incidents efficiently. Squadcast is one such platform that has helped numerous teams respond to critical incidents quicker than ever before.

In fact, Squadcast is packed with a whole lot of easy to use features that help engineers make informed decisions. Now to start with, let’s understand the key benefits that help in alleviating the stress of being on-call.

Benefits of Squadcast

1. Swift Incident Response

For instance, if your monitoring tool sends alerts that you cannot understand at first glance, then your recovery process will be delayed. This would result in increased MTTR and MTTA. With Squadcast, you can add context to the incidents, making it easier to understand and take action. Tagging and Routing are such features that will help you achieve this.

a. Incident Tagging

Responding to incidents becomes much easier when the engineers have enough context regarding the incident. This can be achieved by tagging incidents with relevant information like priority, severity, or alert type within the incoming alert.

This rule-based, auto-tagging system can be established by defining rules on payloads associated with incidents. These tags also help you search for specific incidents and filter a group of incidents on the analytics and incident list pages.

You have the option to tag each incident with a personalized tagging expression as shown in the screenshot below. As soon as an incident is triggered, tags can be automatically attached to it. It also provides insight into the severity levels of incidents. That way, incidents can be better understood.

For example, from the below screenshot, with Datadog's payload, you can define rules in Squadcast to categorize an incident as a high or low priority. This allows you to respond to alerts appropriately.

b. Incident Routing

Usually incident management solutions offer routing as a basic feature that helps teams to direct alerts internally. But in Squadcast, you can customize alert routing based on the rules you define and you will be able to personalize your notification mechanism as well. That way you won't be bothered with irrelevant alerts in your communications channels.

For instance, once the incidents are tagged, you can filter and sort them based on their priority or severity. Then you can route relevant incidents to the right users for resolution. In this case, tagging can make it easier to route low priority incidents to Level-1 support teams and higher priority incidents to Level-2 support teams.

This helps in routing the right alerts to the right responder(s) based on the tags they contain. In this way, it helps greatly in reducing the MTTR of an incident.

2. Reduce Alert Noise, Alert Fatigue, and Operational Toil Drastically

Alert noise is a condition when you're alerted for both critical and non-critical incidents. This is dangerous because it can lead to Alert fatigue and prevents you from responding to critical incidents. With Squadcast you can eliminate this problem completely.

This blog will explain how to optimize your alerts with Squadcast.

a. Incident deduplication (Event Aggregation)

In the case of Alert noise, you can deduplicate events arising from the same source or multiple alert sources (dependent services). As you can see from the below screenshot, by enabling the checkbox at the bottom, you can select other dependent services for the deduplication rule. Now the same incident arising from different alert sources will be grouped into one.

When the first alert for such an incident is triggered, the subsequent alerts for the same incident will be grouped to the original one. But it will be in a ‘triggered’ state across the incident dashboard.

Additionally, you can turn on alert suppression, which will automatically suppress alerts that are not critical. This means you won’t be able to take any action on the alerts since they're in the ‘suppressed’ state on the incident dashboard. You need to be more careful in defining suppression rules because you will no longer be able to act on them.

b. Minimize Operational Toil

Toil is a repetitive set of tasks that makes you feel bored, burned out, and exhausted. In case of incident management, it reduces productivity, affects employee morale, and increases attrition. Click here to read more about how to reduce toil.

Squadcast can significantly reduce an SRE's burden with features such as automatic suppression, deduplication, rule-based routing, escalation policies, and on-call schedules. We have also come up with this unique onboarding checklist that will assist you in getting organized for the on-call process. These personalized options would relieve you from tiring on-call chores.

3. Customize On-Call Operations at Ease

On-call coordination is key to resolving an incident.

Every member in the incident management team must be informed of on-call schedules. Each team member must know the name and phone number of those who are on call at any given time.

Likewise, you need to know who will be on call and at what time. And, if you do not acknowledge an incident on time, it should be escalated to the next on-call engineer.

If this process is not set up properly, it will lead to increased MTTA and MTTR. That’s why on-call schedules and escalation plans play an important role in a smooth incident response process. Squadcast offers 12 layers of personalized escalation policies to alert on-call engineers. That way you will not miss important alerts, and your engineers will not be dismayed either.

a. On-call Escalation Policy

After an incident is triggered, the alerts will be routed based on the defined escalation policy. If a user is not available at that time, the escalation requests are forwarded to the next available user in the escalation policy. This is also called on-call scheduling or on-call rotation.

You will be notified via text messages, emails, and phone calls with our unique 12-layer setup of On-call Escalations. That is, through the platform, the on-call manager can define 12 levels of escalation rules to notify the appropriate user group about an incident. The on-call escalation policy is repeated three times until an incident is acknowledged.

If you are an on-call engineer, you can personalize the notification settings according to your preferences. You can reassign an incident to another user group when necessary. Plus, you can easily create and investigate incidents within Slack.

b. Incident Notes

The platform also comes with an ‘Incident Notes’ feature that allows you to notify a user (on-call engineer) by mentioning their name with ‘@’ in the incident notes panel. So, it fosters ownership of incidents within the team.

c. Incident Postmortem

The platform has incident postmortem templates where you can edit and add details about the incident management processes at ease. While analyzing incidents, postmortems play a crucial role in performing root cause analysis. It helps the team with pre-defined templates to document various findings.

In the above screenshot, you'll see how to edit a predefined incident postmortem template, and the options to update or download it in PDF/MD format. This increases transparency within an organization and all employees can stay informed about the progress of the incident and the various processes that are taken to resolve them.

Final Thoughts

Squadcast, with its customizable features such as contextual tagging, routing, suppression and more, makes it easy for incident management teams to manage and resolve incidents quickly.

By using this intuitive platform, customers have experienced a significant reduction in their MTTR and MTTA. It eliminates alert noise, fatigue, and operational toil, thereby increasing the productivity of the team as a whole.

These are just a handful of benefits discussed in this article. We have a lot of features stacking up in the product roadmap that we'll be sharing with you in the upcoming blogs. So stay tuned! Get started with Squadcast for free and tell us what you think.

Squadcast is an incident management tool that’s purpose-built for SRE. Your team can get rid of unwanted alerts, receive relevant notifications, work in collaboration using the virtual incident war rooms, and use automated tools like runbooks to eliminate toil. Start now for Free.