Better Enterprise Incident Management While Working Remotely: Best Practices from Squadcast

The wide-scale shift to remote work due to Covid-19 and organizations adopting remote work 4 years after the pandemic has made remote incident management the new normal for businesses everywhere. Organizations accustomed to war rooms now rely on collaboration tools like Slack and MS Teams to coordinate teams. This unexpected transition presents unique challenges in enterprise incident management.

Here at Squadcast, we’ve accumulated valuable experience these past months. We’ve identified some best practices that are particularly effective for remote enterprise incident management. While these practices are generally recommended for effective incident management, we believe they’re a crucial starting point for staying on top of issues and preventing major outages, especially in today’s remote work landscape.

Boost Communication and Collaboration

• Solid Communication Plan: Utilize collaboration tools like Slack or MS Teams for incident communication. Have a backup plan in case your primary communication platform goes down. A remote incident response team is essentially a pit crew, working remotely across time zones. Ensure seamless communication to avoid wasting time troubleshooting over phone calls.
• Multi-Tiered Status Pages: Utilize private status pages to keep engineers informed (especially in large teams) while they work on resolving the issue. Public status pages keep customers informed about service disruptions and restoration progress. The recent Slack outage in early 2021 highlights the importance of keeping communication channels open during critical incidents.

Centralize Information Systems

• Information Repository: In a traditional office setting, acquiring system information might involve a quick chat with a colleague. Remote work necessitates a centralized information system with all critical data readily accessible. This eliminates delays and ensures information is available when needed to resolve outages quickly.

Prepare for the Worst with Simulations

• Dry Runs and Simulations: Conduct simulations to assess your team’s remote response to catastrophic failures. This can expose areas for improvement in your incident response strategy.

Leverage Automation to Reduce Toil

• Automation is Key: Automate repetitive tasks such as running scripts, monitoring clusters, scheduling maintenance, and auto-configuring cloud-based virtual machines. Remote work burnout is a real concern, and automation can significantly reduce toil, freeing up your team’s time and energy.
• Automated Runbooks: Detailed runbooks are invaluable during major incidents. Automated runbooks can expedite diagnosing and fixing system outages. Tools like Ansible or Rundeck can be helpful in creating runbooks. Even a basic runbook is superior to manual fixes every time.

Combat Alert Fatigue Proactively

• Fight Alert Fatigue: Remote alert fatigue can be more detrimental than traditional alert fatigue. Effectively configure monitoring tools and adjust alerting thresholds to minimize alert noise. Implement proactive steps to reduce alert noise, such as deduplication rules, event routing, and tagging rules. Enforce mandatory off-days for on-call engineers to prevent burnout.

Coordinate with Developers During Deployments

• Deployment Monitoring: Closely monitor infrastructure during major deployments. Have rollback plans in place in case of unforeseen issues. Since deployments can lead to critical failures, monitor system health during these processes and initiate rollbacks if necessary.

Establish Clear Roles and Responsibilities

• Incident Chain of Command: Define a clear chain of command and designate roles for incident management. This mitigates confusion during time-sensitive and stressful situations, especially when key personnel are unavailable.

Invest in an Enterprise Incident Management Platform

• Dedicated Incident Management Platform: Utilize a dedicated incident management platform to streamline on-call processes with features like escalation policies and alert deduplication rules. Many platforms offer dashboards to track on-call team performance and service quality. Spreadsheets, while once manageable, are no longer sufficient for the clarity and efficiency required in today’s remote work environment. Easy-to-use on-call scheduling features can significantly aid your team in workload planning. Predictable on-call schedules also help prevent burnout.

Leverage Automated Incident Timelines for Continuous Improvement

• Automated Incident Timelines: Following major outages, automated incident timelines provide invaluable data for remote teams to assess the response and identify areas for improvement. At Squadcast, we rely on automated incident timelines to gain real-time insights into incident resolution progress. These timelines are also instrumental in creating post-incident reports (incident postmortems). A detailed record of events allows for a more thorough analysis of your on-call response strengths and weaknesses.

Conclusion

An enterprise incident response team during a major outage resembles a Formula 1 pit crew — a well-coordinated unit working efficiently to resolve issues as quickly as possible. Just like a pit crew, remote incident management teams function best when every member understands their role and responsibilities.

We hope you find these best practices helpful in optimizing your remote enterprise incident management. While this list is not exhaustive, we’d love to hear your thoughts and experiences. What other practices or strategies have you found successful in tackling