Anomaly Detection in Production Systems

Detect unusual metric patterns, latency drifts, and error rate spikes before they become customer-facing incidents — using statistical and ML-based approaches.

🧒 Simple Explanation (ELI5)

Imagine you commute to work every day and the trip takes between 20-35 minutes depending on traffic. One Tuesday it takes 90 minutes. You immediately know something is wrong — a road closure, an accident, something unusual. Your brain detected an anomaly because it knows what "normal" looks like for your commute.

Anomaly detection gives computers this same capability: learn what "normal" looks like for each metric (CPU, response time, error rate), then automatically flag when something falls outside the expected range — before a human would notice.

🔧 Why Anomaly Detection Outperforms Static Thresholds

• Static thresholds are blind to context: CPU > 80% is an alert — but if your batch job runs every night from 2-4am, that's normal. A static threshold pages on-call at 2am unnecessarily.
• Patterns change: A new feature ships. Traffic increases 3x. Now your old 80% CPU threshold fires every deployment. You're constantly chasing threshold adjustments instead of real problems.
• Slow degradation is missed: If your database response time drifts from 20ms to 350ms over 2 weeks, no single minute ever crosses a static threshold — but AI detects the gradual drift.
• Different services have different baselines: Your search API and your batch data processor have completely different normal CPU profiles. Static thresholds can't handle this at scale.

🌍 Real-world Analogy

A cardiologist monitors a patient's heart rhythm over weeks. They know that 55-85 BPM at rest is normal for this patient. An unusual rhythm at 2am — even if technically within normal population range — gets flagged because it's unusual for this specific patient. That's per-entity anomaly detection: personalised baselines for each service, not population-wide rules.

⚙️ Anomaly Detection Approaches

1. Statistical Methods (Fast, Explainable)

• Z-score: Flag values more than N standard deviations from the rolling mean. Fast and explainable, but assumes normal distribution.
• IQR (Interquartile Range): More robust than z-score for skewed data. Flag values outside [Q1 - 1.5×IQR, Q3 + 1.5×IQR].
• Rolling standard deviation: Compute moving window mean and std. Alert when current value > mean + 3×std.

2. Time-Series Models (Handles Seasonality)

• Prophet (Facebook): Decomposes the time series into trend + seasonality + holidays. Detects anomalies as values outside predicted confidence interval. Excellent for metrics with daily/weekly patterns (API traffic).
• ARIMA: Classic time-series forecasting. Predicts next values based on past patterns. Flag when actual exceeds predicted by N standard errors.
• Seasonal Decomposition: Decompose metric into seasonal, trend, and residual components. Anomalies appear in the residual component.

3. ML Models (Complex Multi-variate)

• Isolation Forest: Scores data points by how easily they can be isolated. Effective for multi-variate anomalies (CPU + memory + latency together).
• LSTM Autoencoder: Learns to reconstruct normal patterns. High reconstruction error = anomaly. Good for correlated time-series across services.
• One-Class SVM: Learns the boundary of normal data. Flags anything outside the boundary. Requires careful hyperparameter tuning.

📊 Visual: Anomaly Detection Approaches Comparison

⚡ Kubernetes Integration Flow: Input → AI → Action

How anomaly detection drives automated scaling and alerting in an AKS cluster:

# Step 1: Query Prometheus for recent CPU metrics (last 30 minutes, 1-min resolution)
curl -s 'http://prometheus:9090/api/v1/query_range' \
  --data-urlencode 'query=rate(container_cpu_usage_seconds_total{namespace="prod",pod=~"payment-api.*"}[5m]) * 100' \
  --data-urlencode 'start=now-30m' --data-urlencode 'end=now' --data-urlencode 'step=60' \
  | jq '.data.result[0].values' > /tmp/cpu_metrics.json

# Step 2: Run anomaly detector (Python script from code section)
python3 anomaly_detect.py --input /tmp/cpu_metrics.json --output /tmp/anomaly_result.json

# Step 3a: If anomaly detected, trigger HPA scale-out
ANOMALY=$(jq '.is_anomaly' /tmp/anomaly_result.json)
if [ "$ANOMALY" = "true" ]; then
  kubectl scale deployment/payment-api -n prod --replicas=5
  echo "Auto-scaled payment-api to 5 replicas (CPU anomaly detected)"
fi

# Step 3b: Simultaneously post alert to Slack
curl -X POST "$SLACK_WEBHOOK_URL" \
  -H 'Content-Type: application/json' \
  -d '{"text":"🚨 CPU anomaly on payment-api (score: 0.97). Auto-scaled 3→5 replicas. Investigate: kubectl top pods -n prod"}'

⌨️ Multi-method Anomaly Detection

"""
Production-grade anomaly detection combining statistical and ML methods.
Designed for real-time metric monitoring.
"""
import numpy as np
import pandas as pd
from sklearn.ensemble import IsolationForest
from scipy import stats

class ProductionAnomalyDetector:
    def __init__(self, window_minutes: int = 60):
        self.window = window_minutes
        self.iso_forest = IsolationForest(contamination=0.02, random_state=42)
        self.is_trained = False
        self.baselines = {}

    # ── Statistical: Z-score with rolling window ──────────────────────────────
    def z_score_anomaly(self, series: pd.Series, threshold: float = 3.0) -> pd.Series:
        """Flag points more than `threshold` std deviations from rolling mean."""
        rolling_mean = series.rolling(self.window, min_periods=10).mean()
        rolling_std  = series.rolling(self.window, min_periods=10).std()
        z_scores = (series - rolling_mean) / rolling_std.clip(lower=0.001)
        return z_scores.abs() > threshold

    # ── Statistical: IQR method (robust to outliers in training window) ───────
    def iqr_anomaly(self, series: pd.Series, multiplier: float = 2.5) -> pd.Series:
        """Flag points outside [Q1 - m*IQR, Q3 + m*IQR] computed on rolling window."""
        def iqr_check(window_data):
            if len(window_data) < 10:
                return False
            q1, q3 = np.percentile(window_data, [25, 75])
            iqr = q3 - q1
            lower, upper = q1 - multiplier * iqr, q3 + multiplier * iqr
            return (window_data.iloc[-1] < lower) or (window_data.iloc[-1] > upper)
        return series.rolling(self.window).apply(iqr_check, raw=False).fillna(False).astype(bool)

    # ── ML: Isolation Forest on multi-variate features ────────────────────────
    def train_isolation_forest(self, df: pd.DataFrame, feature_cols: list[str]) -> None:
        """Train on recent normal data."""
        self.iso_forest.fit(df[feature_cols].dropna())
        self.feature_cols = feature_cols
        self.is_trained = True
        print(f"Trained Isolation Forest on {len(df)} samples, {len(feature_cols)} features")

    def ml_anomaly(self, df: pd.DataFrame) -> pd.Series:
        """Return True for rows that Isolation Forest considers anomalous."""
        if not self.is_trained:
            raise RuntimeError("Call train_isolation_forest() first")
        preds = self.iso_forest.predict(df[self.feature_cols].fillna(0))
        return pd.Series(preds == -1, index=df.index)

    # ── Combined: Vote across methods ─────────────────────────────────────────
    def detect(self, df: pd.DataFrame) -> pd.DataFrame:
        """Run all methods; flag row as anomaly if 2+ methods agree."""
        df = df.copy()
        df['z_anomaly']   = self.z_score_anomaly(df['cpu_pct'])
        df['iqr_anomaly']  = self.iqr_anomaly(df['latency_ms'])
        if self.is_trained:
            df['ml_anomaly'] = self.ml_anomaly(df)
        else:
            df['ml_anomaly'] = False

        df['anomaly_votes'] = df[['z_anomaly', 'iqr_anomaly', 'ml_anomaly']].sum(axis=1)
        df['is_anomaly']    = df['anomaly_votes'] >= 2   # require 2/3 consensus

        return df

# ── Usage Example ─────────────────────────────────────────────────────────────
np.random.seed(42)
n = 500
df = pd.DataFrame({
    'cpu_pct':    np.random.normal(45, 8, n).clip(5, 100),
    'latency_ms': np.random.normal(120, 20, n).clip(10, 2000),
    'error_rate': np.random.exponential(0.5, n).clip(0, 50),
})
# Inject anomalies at specific points
df.loc[480:485, 'cpu_pct'] = 92
df.loc[480:485, 'latency_ms'] = 850

detector = ProductionAnomalyDetector(window_minutes=60)
detector.train_isolation_forest(df.iloc[:400], ['cpu_pct', 'latency_ms', 'error_rate'])
results = detector.detect(df)

anomaly_rows = results[results['is_anomaly']]
print(f"Anomalies detected: {len(anomaly_rows)}")
print(anomaly_rows[['cpu_pct', 'latency_ms', 'anomaly_votes']].tail(10))

🧪 Hands-on

1. Run the code above and verify anomalies are detected at rows 480-485 with anomaly_votes >= 2.
2. Inject a slow drift anomaly: gradually increase CPU from row 400 to 490 from 45% to 85%. Which method catches it first? (Z-score should since it uses rolling window)
3. Simulate a Black Friday spike: multiply all values in rows 200-220 by 1.5. This should NOT be flagged as anomalous because all signals spike together. Add a feature check to distinguish coordinated load spikes from individual component failures.
4. Export real Prometheus metric data using promtool query range --start=2h --end=1h 'rate(http_requests_total[5m])' and run your detector against it.
5. Tune the contamination parameter from 0.01 to 0.05 and observe how many more anomalies are detected.

🎮 Try It Yourself

🧠 Debugging Scenario

Problem: Your anomaly detector fires every 7 days, always on Sunday evening at ~9pm UTC, even though there's no real incident.

• Investigation: Examine the metric pattern over 4 weeks — Sunday 9pm shows consistent low-traffic valley (developers return from weekends, US time zone). The model's rolling window doesn't have enough history to learn the weekly pattern and flags the valley as abnormal.
• Root cause: Statistical anomaly detection with a 60-minute rolling window can't learn weekly seasonality.
• Fix option 1: Switch to Prophet or add day-of-week as a conditioning feature. Compute day-specific baselines (separate baseline for Sunday vs. Wednesday).
• Fix option 2: Add minimum duration requirement — an anomaly must persist for 10+ minutes to alert. Valleys that last only 30 minutes get suppressed.
• Fix option 3: Lower the rolling window to capture more historical context: increase from 60min to 10080min (1 week).

🎯 Interview Questions

🌐 Real-world Usage

Azure Monitor uses per-resource dynamic thresholds — it learns a 4-week baseline for each metric on each specific resource, accounting for time-of-day and day-of-week patterns. Netflix uses a multi-layer anomaly detection stack: statistical models for real-time detection (low latency), ML models for seasonal pattern anomalies (hourly retrain), and LLM-based analysis for complex correlated failures. Dynatrace Davis AI uses causal AI to not just detect anomalies but automatically determine which anomaly caused which downstream impact.

📝 Summary

Effective production anomaly detection combines multiple approaches: statistical methods for speed and explainability, time-series models for seasonality awareness, and ML models for multi-variate correlation. Always use ensemble voting (2+ methods) and integrate a maintenance calendar to suppress expected spikes. The goal isn't zero false positives — it's raising the signal-to-noise ratio so engineers trust and act on alerts.