How to Keep Track of Your On-Call Responsibilities

An Easy Way to Manage On-Call Rotations

On-call engineers are the initial line of defense when a system outage occurs, ensuring that critical services impacting customers are quickly identified and resolved. This blog post will explore some of the use cases that can help you avoid pitfalls in your organization’s on-call rotation process. You’ll also learn how to get free access to an on-call onboarding checklist.

The Importance of On-Call Scheduling

A common first step for organizations towards achieving reliability for their customers and users is to implement an on-call rotation. In an on-call rotation, on-call engineers act as the first responders, ensuring that outages impacting customers are addressed swiftly.

On-Call Scheduling Software

We, at Squadcast, use our own on-call rotation system built on our platform. This system supports many customizable on-call rotation options, including:

• Follow-the-sun rotations
• Daily rotations
• Weekly rotations
• Split-shift rotations

Our system also allows you to create multiple scheduling layers within a single schedule.

Benefits of On-Call Scheduling Scripts

On-call scheduling scripts can help you view on-call responsibilities. These scripts can help you find out who is on-call at a given time and which Slack channel to use to send on-call notifications for a particular schedule.

How the Script Works

The script retrieves a list of people who are on-call for a specified Squadcast Schedule across all the shifts, at the time the script is run. It then sends this list to a Slack channel.

Script Inputs

The script requires a configuration JSON file as a command-line argument. The configuration file looks something like this:

JSON

{
"REFRESH_TOKEN": "Squadcast Refresh Token which will be used to be make requests to Public APIs",
"SCHEDULE_NAME": "Name of the schedule for which we are getting the on-call people",
"SLACK_WEBHOOK_URL": "Slack incoming webhook URL for the channel to which we want to send the on-call notification"
}

Steps to Use the Script

1. Clone the Repo

• Open a terminal window and run the following command to clone the repository: git clone git@github.com:SquadcastHub/who-s-oncall-slack.git

1. Getting the Refresh Token

• If you are an Account Owner or Admin of the Squadcast account, follow these steps to obtain a refresh token. If you are a Stakeholder or User, contact your Account Owner or Admin to get the token.
• Log in to your Squadcast Dashboard and go to the profile page.
• You’ll find an API Access Token here. If you haven’t already created one, click on “Generate new API token.”
• Copy the generated token to the value corresponding to the “REFRESH_TOKEN” key in the “params.json” file.

1. Getting the Schedule Name

• Select Schedules from the Sidebar within the platform.
• On the right side of the screen, you can see a list of the different schedules you’ve created. Copy the name of the Schedule for which you want to get the list of on-call people to the “params.json” file.

1. Getting the Slack Webhook URL

• Open your Slack app.
• Scroll through the sidebar on the left to get to the “Apps” section. Click on the “+” button next to “Apps.”
• Search for “Webhook” in the search box. Click on the “Add” button of the “Incoming WebHooks.”
• You’ll be redirected to a webpage. Click on the “Add to Slack” button.
• You’ll be directed to a screen where you’ll be asked to choose which Slack channel the webhook will send messages to. Click on “Add Incoming WebHook Integration” once done.
• In the next screen, you’ll get your Webhook URL. Copy it to the “params.json” file.

Building and Running the Script

Now that you have your “params.json” file ready, you’ll need to build the script and configure cron.

Note: You’ll need Golang installed on your system to build this script.

• If you don’t already have Golang installed, you can download it here
• To build the script, navigate to the script’s directory and run the following command:

go build

• Once the build is successful, you’ll see a binary file named “oncall” in the current directory. To run this binary, you’ll need to specify the path to the “params.json” file containing all the parameters configured in the previous steps. The command to run would be:

./oncall <path_to_params.json>

Setting Up Cron Job in Linux

The script itself doesn’t handle scheduling its execution. You’ll need to set up a cron job to automate this process.

• Let’s assume that the on-call hand-off at your organization happens every Monday at 12 PM. To run the script at this time, follow these steps:
• Open a terminal and run the following command:

crontab -e

• A text editor will open in your terminal. Paste the following line into the file and save it:

0 12 * * 1 <path_to_binary> <path_to_params.json>

Replace the following placeholders in the command:

• <path_to_binary>: Absolute path to the script binary (e.g., /home/oncall).
• <path_to_params.json>: Absolute path to the params.json file (e.g., /home/params.json).

For more details on using the crontab command, you can check out its manual page by running the following command:

man crontab

You can also refer to these resources for additional help with cron jobs:

• https://www.fastcomet.com/kb/setting-up-cron-jobs-in-cpanel
• https://crontab.guru/

Important Note Regarding Cron Job Frequency

• Make sure you don’t call the script very frequently to avoid overwhelming the system. As a general rule, avoid running the script more than once within an hour.

Once the cron job is set up, you’ll receive notifications on the designated Slack channel about the new on-call users whenever the script runs.

Additional Considerations

• Can I receive notifications on other chat platforms like Google Hangouts or Microsoft Teams?

Yes! You can modify the script to work with other chat platforms. Simply replace the NotifySlack function with your own functions named NotifyHangouts or NotifyTeams, following the respective chat platform's API documentation for setting up notifications.

• How can I group usernames by shifts?

The current API doesn’t support grouping usernames by shift names. However, the script will be updated to reflect this functionality when it’s incorporated into the API.

• Can I get on-call users for multiple schedules?

Absolutely! You can create multiple params.json files with different configurations and set up the cron job to run the script with each unique configuration.

For instance, if you have two params.json files with absolute paths /home/params1.json and /home/params2.json and the binary at /home/oncall, you can add the following lines to your crontab file:

0 12 * * 1 /home/oncall /home/params1.json
0 12 * * 1 /home/oncall /home/params2.json

This configuration allows you to:

• Send notifications for various schedules to the same or different Slack channels.
• Schedule crons for different schedules to run at the same or different times.

Conclusion

By implementing on-call rotations and using on-call scheduling software, you can ensure your team is prepared to handle critical issues efficiently. This blog post provided a helpful overview of on-call rotations, on-call scheduling software, and on-call rotation generators. It also offered a solution for managing on-call responsibilities using Squadcast’s on-call scheduling system and a script to automate on-call notifications.

Squadcast is a Popular Pagerduty Alternative, Incident Management tool that’s purpose-built for SRE. Get rid of unwanted alerts, receive relevant notifications and integrate with popular ChatOps tools. Work in collaboration using virtual incident war rooms and use automation to eliminate toil.