Incident Management, On-call and Escalation
ELI5 Explanation
When production breaks, everyone needs a clear playbook: who leads, who communicates, who fixes, and when to call more help.
Technical Explanation
Incident management uses severity levels, command roles, communication cadence, and escalation policies. Good on-call systems protect engineers with clear handoffs, fair schedules, and actionable alerts. Incident response quality is measured through MTTD, MTTA, and MTTR.
Visual
Hands-on Commands
kubectl get pods -A --field-selector=status.phase!=Running
kubectl describe pod -n
kubectl logs -n --previous
kubectl rollout undo deploy/ -n Debugging Scenario
At 2 AM, checkout latency spikes and error rate grows. First responder creates incident bridge, tags severity, assigns commander, and posts customer status every 15 minutes. Team mitigates by traffic shifting and rollback while parallel team investigates root cause.
Beginner
- What is an incident commander?
- Why do severity levels matter?
- What is the purpose of on-call rotations?
- What does MTTR mean?
- Why are status updates important?
Intermediate
- How do you design escalation policy tiers?
- How do you reduce alert fatigue on-call?
- When do you declare a major incident?
- How do you split mitigation and diagnosis streams?
- What data should be captured during incident timeline?
Scenario-based
- Primary on-call is unavailable. What failsafe process do you implement?
- Two teams blame each other during outage. How do you keep response effective?
- Incident gets fixed but keeps recurring weekly. What do you enforce post-incident?
- Management asks for ETA too early. How do you communicate uncertainty?
- Global outage starts in one region. How do you trigger escalation by blast radius?
Real-world Use Case
A fintech team introduced incident command templates and quarterly game days. MTTR dropped 40%, and customer comms became predictable during outages.
Summary
Effective incident handling requires structured roles, disciplined communication, and fast mitigation. Next, you will convert incident data into lasting improvements via RCAs and blameless postmortems.