Incident Summarization and Intelligent Alerts

Auto-generate structured incident summaries and context-rich alerts using LLMs — so on-call engineers get a clear situation brief in seconds instead of spending 20 minutes reading logs.

🧒 Simple Explanation (ELI5)

Imagine you're called at 2am about a problem at a factory. A traditional alert says: "Alarm 409 triggered in Zone 7." You have no idea what that means. An intelligent alert says: "The packaging machine in Zone 7 stopped working at 1:58am. This has happened twice before — both times a conveyor belt sensor caused it. Maintenance runbook: check sensor cable in panel B-3." You know exactly what happened, why, and what to do — all before you've got out of bed.

🔧 Why Incident Summarization Reduces MTTR

• Context switching cost: Engineers wake up to an alert and spend 15-20 minutes just understanding what's happening before they can act. AI-generated summaries cut this to 30 seconds.
• Information scatter: Incident data lives in logs, metrics, traces, Slack messages, and deployment history. Humans manually correlate all this. LLMs do it in one call.
• Knowledge capture: Senior engineers know that a specific error pattern means DB pool exhaustion. AI learns this association from historical incidents and embeds it in every summary.
• Runbook linking: AI maps error patterns to relevant runbooks automatically, so new engineers know exactly where to go.
• Handover notes: When incidents span shifts, AI writes the handover brief automatically, ensuring nothing is lost between teams.

🌍 Real-world Analogy

A hospital Emergency Department uses a triage nurse who, within 3 minutes of a patient arriving, produces a structured brief for the attending physician: chief complaint, vital signs, relevant medical history, suspected condition, and immediate actions needed. The physician can make an informed decision in 60 seconds instead of starting from scratch. AI incident summarization is exactly this triage brief — for production systems.

⚙️ How AI Incident Summarization Works

Step 1: Context Aggregation

Before calling the LLM, collect all available context:

• Top error log clusters (from log analysis pipeline)
• Anomalous metric values (from anomaly detector)
• Recent deployments in the last 2 hours (from CI/CD events)
• Service dependency graph (which services call this service)
• Similar past incident IDs (from vector search on incident history)

Step 2: Structured Summarization Prompt

Engineering the prompt to get consistent, structured output is critical:

• Specify exact output schema (JSON with required fields)
• Include schema examples to prevent format drift
• Use temperature 0.1 for deterministic output — never high temperature for triage
• Cap input context to avoid token limit issues

Step 3: Alert Enrichment

Attach the AI summary to the incident ticket with:

• Severity, root cause hypothesis, affected users/services
• Linked runbook URL
• Similar past incidents (with resolution times)
• Confidence score (flag low confidence for human review)

📊 Visual: Intelligent Alert Architecture

⚡ Kubernetes Integration Flow: Input → AI → Action

How AI incident summarization runs during a multi-service alert storm in AKS:

# Collect context from AKS before calling the LLM

# 1. Get recent Kubernetes events (deployment failures, pod restarts, OOMKills)
kubectl get events -n prod --sort-by='.lastTimestamp' --field-selector type=Warning \
  | tail -20 > /tmp/k8s_events.txt

# 2. Get Helm release history (was there a recent upgrade?)
helm history payment-api -n prod --max 3 > /tmp/helm_history.txt

# 3. Get current pod states
kubectl get pods -n prod -l app=payment-api -o wide > /tmp/pod_status.txt

# 4. Feed all context into the Python summarisation script
python3 generate_incident_brief.py \
  --service payment-api \
  --k8s-events /tmp/k8s_events.txt \
  --helm-history /tmp/helm_history.txt \
  --pod-status /tmp/pod_status.txt \
  | curl -X POST "$PAGERDUTY_EVENTS_URL" \
    -H 'Content-Type: application/json' \
    --data-binary @-

⌨️ AI Incident Summarization with Azure OpenAI

"""
AI incident summarization pipeline.
Aggregates multi-source context and generates a structured incident brief.
"""
import os
import json
import datetime
import requests

AZURE_ENDPOINT = os.getenv("AZURE_OPENAI_ENDPOINT")
API_KEY        = os.getenv("AZURE_OPENAI_KEY")
DEPLOYMENT     = "gpt-4"

def create_incident_brief(
    service: str,
    log_clusters: list[dict],
    anomalous_metrics: list[dict],
    recent_deployments: list[dict],
    similar_incidents: list[dict]
) -> dict:
    """
    Generate a structured incident brief from multi-source context.
    Returns validated JSON matching the IncidentBrief schema.
    """
    # Build context string (keep concise to save tokens)
    log_summary = "\n".join([
        f"- [{c['count']}x] {c['template'][:120]}" for c in log_clusters[:5]
    ])
    metric_summary = "\n".join([
        f"- {m['name']}: {m['current_value']} (baseline: {m['baseline_value']}, {m['deviation_pct']:+.0f}%)"
        for m in anomalous_metrics[:3]
    ])
    deploy_summary = "\n".join([
        f"- {d['service']} v{d['version']} deployed {d['minutes_ago']}min ago by {d['author']}"
        for d in recent_deployments[:2]
    ]) or "No deployments in the last 2 hours"
    past_incidents = "\n".join([
        f"- INC-{i['id']}: {i['title']} (resolved in {i['resolution_minutes']}min)"
        for i in similar_incidents[:2]
    ]) or "No similar past incidents found"

    prompt = f"""You are a senior SRE performing incident triage. Analyse the following data for service '{service}'.
Return ONLY valid JSON matching this exact schema — no markdown, no explanation:
{{
  "severity": "critical|high|medium|low",
  "summary": "2-3 sentence plain-English description of what is happening",
  "root_cause_hypothesis": "most likely root cause in one sentence",
  "affected_services": ["list of affected service names"],
  "user_impact": "description of what users are experiencing",
  "immediate_action": "the single most important immediate action",
  "runbook_hint": "which runbook or procedure is most relevant",
  "confidence": 0.0
}}

TOP ERROR LOG CLUSTERS:
{log_summary}

ANOMALOUS METRICS:
{metric_summary}

RECENT DEPLOYMENTS:
{deploy_summary}

SIMILAR PAST INCIDENTS:
{past_incidents}"""

    response = requests.post(
        f"{AZURE_ENDPOINT}/openai/deployments/{DEPLOYMENT}/chat/completions?api-version=2024-06-01",
        headers={"api-key": API_KEY, "Content-Type": "application/json"},
        json={
            "messages": [
                {"role": "system", "content": "You are a production incident triage bot. Output structured JSON only."},
                {"role": "user",   "content": prompt}
            ],
            "max_tokens": 400,
            "temperature": 0.1
        },
        timeout=30
    )
    response.raise_for_status()
    raw_output = response.json()["choices"][0]["message"]["content"]

    # Validate and parse JSON output
    try:
        brief = json.loads(raw_output)
    except json.JSONDecodeError:
        # Retry with explicit JSON extraction if LLM wrapped in markdown
        import re
        json_match = re.search(r'\{.*\}', raw_output, re.DOTALL)
        brief = json.loads(json_match.group()) if json_match else {"error": "JSON parse failed", "raw": raw_output}

    brief["generated_at"] = datetime.datetime.utcnow().isoformat()
    brief["service"] = service
    return brief

# ── Example usage ─────────────────────────────────────────────────────────────
sample_brief = create_incident_brief(
    service="payment-api",
    log_clusters=[
        {"count": 847, "template": "DB connection refused <IP>:5432 after timeout"},
        {"count": 423, "template": "Upstream payment-service returned <NUM> status code"},
        {"count": 112, "template": "Connection pool exhausted - max <NUM> connections"},
    ],
    anomalous_metrics=[
        {"name": "db_connection_time_ms", "current_value": 4200, "baseline_value": 45, "deviation_pct": 9233},
        {"name": "error_rate_pct",        "current_value": 23.4, "baseline_value": 0.3, "deviation_pct": 7700},
        {"name": "cpu_pct",               "current_value": 18,   "baseline_value": 44,  "deviation_pct": -59},
    ],
    recent_deployments=[
        {"service": "postgres-proxy", "version": "2.4.1", "minutes_ago": 18, "author": "terraform-ci"}
    ],
    similar_incidents=[
        {"id": "4521", "title": "DB pool exhaustion after proxy upgrade", "resolution_minutes": 23}
    ]
)
print(json.dumps(sample_brief, indent=2))

🧪 Hands-on

1. Set up Azure OpenAI credentials and run the create_incident_brief() function with the sample data above.
2. Verify the output severity is "critical" and the root cause hypothesis mentions the database proxy deployment.
3. Test the JSON parsing resilience: modify the prompt to occasionally return markdown-wrapped JSON (```json{...}```) and verify the regex fallback catches it.
4. Integrate with a PagerDuty or Opsgenie API: when the brief is generated, automatically create a P1 incident if severity is "critical" with the brief as the incident description.
5. Build a feedback mechanism: add a "Was this summary correct? 👍 👎" button to your Slack alert. Track the ratio over time to measure LLM quality.

🎮 Try It Yourself

🧠 Debugging Scenario

Problem: AI incident summary says "Root cause: DNS resolution failure" but the real incident was a database connection pool exhaustion. The summary is confidently wrong and engineers wasted 10 minutes investigating DNS.

• Root cause 1: The context aggregator fed the LLM error logs from a different service that also had DNS issues at the same time (coincidental correlation). The LLM correctly summarised the context it was given, but the context itself was wrong (included irrelevant services).
• Root cause 2: The similar past incidents search returned an old DNS incident because the error message contains similar tokens. Semantic similarity was misleading.
• Fix: 1) Scope log collection to the specific service and its direct dependencies only. 2) Add deployment context to help LLM distinguish temporal correlation from causal relationship. 3) Add a post-incident feedback loop: when engineers mark the AI root cause as "incorrect," use it to improve the vector search and prompting. 4) Display confidence score prominently — a 0.58 confidence DNS hypothesis should be visually flagged as speculative.

🎯 Interview Questions

🌐 Real-world Usage

PagerDuty's AIOps feature uses ML to group related alerts into a single incident and generates an automated triage note. Microsoft's Azure Monitor uses AI-powered "insight explanations" that describe in plain English why a smart detection alert fired. Atlassian's Jira Service Management uses AI to generate incident summaries from linked monitoring data. LinkedIn reduced average time-to-acknowledge by 40% after implementing automated incident context enrichment.

📝 Summary

AI incident summarization aggregates multi-source context (logs, metrics, deployments, past incidents), sends it to an LLM with a structured prompt, and produces a brief that on-call engineers can act on immediately. Key engineering requirements: structured JSON output, low temperature for determinism, confidence scoring, graceful degradation, and a feedback loop to improve quality over time.