Alert Design
Design three alerts for a loan or fraud model: one operational alert, one drift alert, and one business KPI alert. For each, state who gets notified, what investigation should start, and whether traffic should be reduced immediately.
Model Monitoring, Drift, and Observability
Monitor not only whether the service is up, but whether the model is still correct, fair, stable, and aligned with business outcomes.
🧒 Simple Explanation (ELI5)
A weather app can open quickly and still be wrong about tomorrow's rain. Model monitoring checks both whether the app works and whether the predictions still make sense.
🔧 Why Do We Need It?
- Data changes: production inputs rarely stay identical to training data forever.
- Concepts shift: the relationship between input and outcome can change over time.
- Business damage can be silent: a model may degrade before infrastructure alarms fire.
- Observability drives retraining decisions: teams need evidence, not guesses.
🌍 Real-world Analogy
A pilot does not only monitor whether the aircraft engine is running. They also monitor altitude, fuel, weather, direction, and landing path. Good ML monitoring watches more than uptime for the same reason.
⚙️ Technical Explanation
Model observability combines system metrics and model metrics. Infrastructure signals include latency, throughput, error rate, CPU, memory, and request distribution. ML-specific signals include input drift, target drift, concept drift, prediction distribution shifts, label delay, fairness drift, and business KPI impact. A mature design also monitors data quality such as null rates, category explosions, and schema changes.
Drift does not always mean retrain immediately. Sometimes it signals upstream data issues or segment behavior changes that require investigation first. Monitoring should therefore support diagnosis, not only alerting.
📊 Visual Representation
Observability Stack
Signals
Latency
Input drift
Prediction shift
Business KPI
→
📈 Monitor + Alert
→
🔍 Investigate / Retrain
⌨️ Commands / Syntax
python
from scipy.stats import ks_2samp
reference = train_df["income"].dropna()
current = prod_df["income"].dropna()
stat, p_value = ks_2samp(reference, current)
print({"ks_stat": round(stat, 3), "p_value": round(p_value, 5)})
kusto
// Example: track inference failures by model version AppTraces | where Message contains "prediction_failed" | summarize failures=count() by ModelVersion, bin(TimeGenerated, 15m)
💼 Example (Real-world Use Case)
A credit approval model remains technically healthy, but approval rates for one customer segment rise sharply after a policy change. Monitoring catches the prediction distribution shift and segment disparity before it becomes a compliance problem. The team pauses promotion and starts a retraining investigation.
🧪 Hands-on
- List three infrastructure metrics and three ML-specific metrics you should monitor for one production model.
- Define which ones create alerts and which ones create investigation tickets.
- Write one example of a drift signal that should not automatically trigger retraining.
- Choose one business KPI that must be reviewed alongside prediction metrics.
🎮 Try It Yourself
🎮
🐛 Debugging Scenario
Problem: the monitoring system reports severe drift, but model quality appears unchanged.
- Root cause 1: a harmless marketing campaign changed traffic mix without changing decision quality.
- Root cause 2: the drift baseline was outdated or too narrow.
- Root cause 3: one noisy feature dominated the alert threshold.
- Fix: inspect segment-level impact, refresh baselines, and prioritize drift signals tied to business or label outcomes.
🎯 Interview Questions
Beginner
What is model drift?
Model drift is when live data or live behavior changes enough that model performance may degrade.
Why is uptime not enough for model monitoring?
Because a model can be fast and available while still making bad decisions.
What is input drift?
Input drift is when the distribution of live features changes compared with training or baseline data.
What is concept drift?
Concept drift is when the relationship between inputs and outputs changes over time.
Why monitor prediction distribution?
Because sudden shifts can reveal behavior changes even before labels arrive.
Intermediate
What is the difference between data drift and concept drift?
Data drift changes the input distribution; concept drift changes how inputs map to outcomes.
Why are delayed labels a monitoring challenge?
You cannot measure true model accuracy immediately when the real outcome arrives much later.
Why should monitoring be segment-aware?
Because average metrics can hide major failures in specific regions, customer types, or channels.
When should drift trigger retraining automatically?
Only when the drift signal is trusted, sustained, and tied to degraded business or quality outcomes.
What is the biggest observability anti-pattern in MLOps?
Monitoring infrastructure health while ignoring prediction quality and business impact.
Scenario-based
A model's latency is normal, but approval rates changed by 20%. What do you do?
Investigate prediction shift, segment behavior, policy changes, and whether a rollback or traffic reduction is needed.
A drift alert fires every holiday season. How do you improve it?
Use seasonal baselines and tie alerts to business impact instead of raw drift alone.
Labels arrive 30 days late. How do you monitor before then?
Monitor input drift, prediction distributions, proxy business signals, and delayed accuracy backfills.
An ops team wants every drift alert to auto-retrain. What is wrong with that?
Some drift is harmless or temporary; automatic retraining can create churn, cost, and unstable releases.
How do you explain observability value to leadership?
It reduces blind spots, catches business-impacting degradation earlier, and lowers the cost of bad decisions.
🌐 Real-world Usage
Recommendation, fraud, risk, and pricing platforms all depend on strong observability because the most expensive failures are often silent quality failures rather than obvious outages.
📝 Summary
Monitoring tells you whether the model is still useful, not just whether the server is still running. Strong MLOps observes data, predictions, infrastructure, and business impact together.