Incident Response Guide
A practical reliability guide for small teams building incident workflows.
Incident Response Guide from Sandglass: practical guidance for moving from ad-hoc outage reactions to a repeatable owner, channel, timeline, and recovery process.
What this guide covers
This guide focuses on moving from ad-hoc outage reactions to a repeatable owner, channel, timeline, and recovery process. The goal is to make the operating decision clear before a stressful incident forces the team to improvise.
- One incident lead keeps decisions and communication coherent.
- Severity decides how heavy the response should be.
- A recorded timeline makes the post-incident review honest.
How Sandglass supports the practice
Route production incidents to the team channel, assign one incident lead, record decisions as they happen, and review the alert after recovery. Sandglass supports the continuous side of this work with checks, incidents, alert routing, and public status visibility.
- Back the practices here with HTTP, ping, TCP, content, SSL certificate, and heartbeat checks.
- Route incidents to email, Slack webhook channels, and generic webhooks so the right people respond fast.
- Use a public status page to keep customers informed while the team works the incident.
Common mistakes to avoid
Incident process fails when every alert becomes a meeting. Use severity, ownership, and recovery criteria so the response matches the real customer impact.
Implementation checklist
Step 1: Start from customer impact
Decide which failures in this topic actually reach customers before adding any monitoring.
Step 2: Choose one signal per risk
Match each risk to a single HTTP, content, TCP, SSL certificate, or heartbeat check instead of stacking duplicates.
Step 3: Assign an owner and a channel
Give each alert one owner and one destination — email, a Slack webhook, or a generic webhook.
Step 4: Review after real incidents
Revisit intervals, thresholds, and ownership once a real incident shows what was missing.