Follow the real production loop every ML system lives in: train, validate, deploy, monitor, and retrain when evidence says the model is stale.
Training teaches the model. Validation checks whether it is good enough. Deployment gives it real work. Monitoring watches whether it stays useful. Retraining updates it when reality changes.
It is like hiring a pilot. Training school is not enough. The pilot must pass checks, fly under controlled conditions, be monitored in live service, and be retrained when procedures or aircraft change.
The lifecycle starts with versioned code, curated data, and a reproducible environment. Training produces one or more candidate models. Validation then applies gates such as metric thresholds, latency checks, data quality checks, and bias checks. Approved models are registered, deployed to staging or production, and observed in live traffic. Monitoring collects infrastructure metrics, prediction behavior, input drift, and business KPIs. Retraining is triggered when the live model is genuinely stale, not when the release process itself is broken.
Train builds a candidate. Validate decides whether it is promotable. Deploy releases it safely. Monitor checks whether it remains healthy. Retrain creates a new candidate only when real evidence says the current one is stale.
Use rollback for bad releases, serving errors, or wrong artifacts. Use retraining for legitimate model staleness caused by drift, new labels, or changed business patterns.
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
jobs:
train:
type: command
command: python train.py --data data/train.csv
validate:
type: command
command: python validate.py --model outputs/model.pkl --min_auc 0.84
register:
type: command
command: python register.py --name churn-model --path outputs/model.pkl
az ml job create --file pipeline.yml az ml model list --name churn-model az ml online-endpoint list
A telecom churn model is retrained weekly. If AUC drops below 0.84 or precision on high-risk accounts regresses, validation blocks release. Approved models are registered in Azure ML, deployed to staging, and shifted into production with a canary. Monitoring checks live conversion and latency. If the new release is broken, the team rolls back. If the live model is stable but stale, they retrain.
Take a simple model such as house-price prediction and describe how you would train it, validate it, deploy it, monitor it, and retrain it. Then identify one failure where rollback is correct and another where retraining is correct.
Problem: validation passes, deployment succeeds, but customers report inconsistent predictions across regions.
It should be validated, deployed safely, monitored in production, and retrained only when justified.
Training creates candidates. Validation decides whether a candidate is safe enough to promote.
Because live data, traffic, and business behavior change after release.
When monitoring shows the model is stale or new trustworthy labels justify a rebuild.
Rollback fixes a bad release. Retraining addresses a stale but technically healthy model.
Because the offline dataset may not represent live traffic or business segments accurately.
Metric thresholds, schema validation, latency checks, lineage, and approval policy belong there.
They reduce risk by exposing only a portion of traffic first.
It should include drift, prediction distributions, error rates, and business KPI impact.
Because you can automate lower-quality models into production and hide the real root cause of incidents.
Pause or roll back the rollout, inspect segment-level behavior, and do not treat offline lift as proof of business success.
Not automatically. First check whether the drift is benign, temporary, or correlated with business degradation.
It adds cost and release risk without proving that the current model is actually stale.
Because the problem may be serving, packaging, or routing, not model staleness.
It is the control system that turns ML from a one-off experiment into a reliable production capability.
Recommendation, fraud, forecasting, and document-processing systems all follow this same loop. High-performing teams make every stage visible, measurable, and auditable.
The ML lifecycle is not train once and forget. It is a controlled production loop that keeps model releases safe and model value current.