Incident Management¶
An incident is any event that hurts users or the business. A clear process lets people act calmly instead of improvising under pressure.
Lifecycle¶
- Detect: an alert or a user report.
- Triage: assess impact and set a severity.
- Mitigate: stop the harm first. Rollback, fail over or scale. Root cause comes later.
- Communicate: keep stakeholders informed on a regular rhythm.
- Resolve: confirm recovery against the SLI.
- Learn: hold a postmortem.
Roles¶
| Role | Job |
|---|---|
| Incident commander | Coordinates and decides. Does not debug. |
| Operations lead | Investigates and applies fixes. |
| Communications lead | Updates stakeholders and customers. |
| Scribe | Keeps the timeline. |
Small teams combine roles, but someone must own coordination.
Example severity levels¶
Adapt these to your organisation:
| Level | Meaning |
|---|---|
| SEV1 | Major outage or data loss. All hands. |
| SEV2 | Significant degradation for many users. |
| SEV3 | Limited impact. Handle in working hours. |
First ten minutes¶
- Open one channel and name an incident commander.
- State impact and the time it started.
- Check recent changes first. They explain most incidents.
- Prefer rollback to a risky live fix.
- Post an update, and then keep to a schedule.
For hands-on diagnosis see Troubleshooting.
Healthy on-call¶
Keep rotations sustainable, make handoffs explicit, and keep runbooks next to the alerts they explain. Every page should be actionable, or the alert should change.