AdvancedLesson 9 of 16

Multi-Stage Pipelines

Design promotion flows that build once, validate once, and deploy safely across Dev, QA, UAT, and Production with approvals, environment checks, and reusable stage patterns.

🧒 Simple Explanation (ELI5)

A multi-stage pipeline is like an airport journey with checkpoints. First you check in, then security verifies you, then boarding starts, then you reach the destination. You do not restart the entire trip at every gate. You keep moving the same traveler forward after each checkpoint succeeds.

That is what a multi-stage pipeline does with software: build once, test once, then promote the same proven output through higher environments instead of rebuilding every time.

🔧 Technical Explanation

Why Multi-Stage Matters

Build Once, Promote Many
Build
Test
Dev
QA
Prod

Stages, Jobs, Deployments, and Environments

ObjectPurposeTypical Example
StageHigh-level phaseBuild, Test, Deploy_Prod
JobAgent execution unitRun unit tests on Ubuntu
Deployment jobEnvironment-aware deployment recordDeploy to AKS production
EnvironmentTarget with checks and historystaging, production

Reference YAML Structure

yaml
trigger:
- main

stages:
- stage: Build
  jobs:
  - job: BuildAndTest
    pool:
      vmImage: ubuntu-latest
    steps:
    - script: echo "lint, test, package"
    - publish: drop
      artifact: app

- stage: Deploy_Dev
  dependsOn: Build
  condition: succeeded()
  jobs:
  - deployment: DeployDev
    environment: dev
    strategy:
      runOnce:
        deploy:
          steps:
          - download: current
            artifact: app
          - script: echo "deploy to dev"

- stage: Deploy_Prod
  dependsOn: Deploy_Dev
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  jobs:
  - deployment: DeployProd
    environment: production
    strategy:
      runOnce:
        deploy:
          steps:
          - download: current
            artifact: app
          - script: echo "deploy to prod"

Approvals and Checks

Approvals do not usually live inside the YAML itself. They are attached to the environment in Azure DevOps. This is deliberate because sensitive deployment rules should not be bypassed by simply editing the pipeline file.

Separation of Duties

For high-risk environments, keep the deployment path in YAML but enforce approvals and checks at the environment layer. That gives you versioned pipeline logic plus centrally governed release controls.

🛠️ Hands-on

Build a Dev-to-Prod Promotion Pipeline

  1. Create environments named dev, qa, and production.
  2. Add a production approval check requiring an operations approver.
  3. Publish one artifact in the Build stage.
  4. Deploy that same artifact through each stage.
  5. Use stage conditions so production only runs from main.

Example with AKS and Helm

yaml
- stage: Deploy_Staging
  jobs:
  - deployment: StagingDeploy
    environment: staging
    strategy:
      runOnce:
        deploy:
          steps:
          - script: |
              helm upgrade --install webapp ./charts/webapp \
                --namespace staging \
                --set image.tag=$(Build.BuildId)

- stage: Deploy_Production
  dependsOn: Deploy_Staging
  jobs:
  - deployment: ProdDeploy
    environment: production
    strategy:
      runOnce:
        deploy:
          steps:
          - script: |
              helm upgrade --install webapp ./charts/webapp \
                --namespace production \
                --set image.tag=$(Build.BuildId)

This connects directly to AKS and Helm without repeating cluster basics. The difference here is promotion logic, not Kubernetes theory.

🐛 Debugging Scenarios

Scenario 1: Stage Skips Unexpectedly

Scenario 2: Production Never Starts

Scenario 3: Artifact Missing in Later Stage

📋 Interview Questions

Beginner

What is a multi-stage pipeline?

A single pipeline containing multiple sequential or conditional stages such as build, test, and deployment, usually promoting one artifact through several environments.

Why is build-once-deploy-many important?

It ensures the exact same tested output is what gets deployed later, reducing drift and increasing release confidence.

What is an environment in Azure DevOps?

An environment is a deployment target construct that stores history, approvals, checks, and associated resources.

What is the difference between a normal job and a deployment job?

A deployment job is linked to an environment and gives richer deployment tracking and governance controls than a plain job.

Can stages run in parallel?

Yes, if you do not create dependencies between them and your pipeline design allows parallel execution.

Intermediate

Where should approvals live: YAML or environment settings?

For strong governance, approvals should live on the environment so they cannot be removed casually in a pull request.

How do conditions control release flow?

Conditions let you gate stages by branch, tag, run reason, prior stage status, or custom variables, making promotion paths explicit and predictable.

Why are deployment jobs useful for AKS releases?

They tie each deployment to an environment history, approvals, and checks, which is valuable for auditing and change control around cluster releases.

What is a common anti-pattern in multi-stage design?

Rebuilding in every stage instead of promoting one immutable artifact. That defeats the whole point of consistent promotion.

How do you keep multi-stage YAML maintainable?

Use templates, consistent stage naming, shared variables, and keep environment-specific differences in small, explicit inputs rather than copy-pasting whole stages.

Scenario-Based

A deployment reached staging but production was skipped. How do you investigate?

I check stage conditions, environment approvals, branch filters, and upstream stage status. Then I confirm whether skipping was intentional or caused by a misconfigured condition.

How would you explain multi-stage pipelines to an auditor?

They provide a traceable chain from source commit to built artifact to each environment deployment, with policy checks and approval records at higher-risk environments.

A team wants one YAML file per environment instead of one multi-stage file. What trade-off do you discuss?

Separate files can simplify local ownership, but they often duplicate logic and weaken end-to-end traceability. A single multi-stage pipeline keeps promotion history clearer if the release model is shared.

How do you combine blue-green or canary with multi-stage pipelines?

Use separate deployment stages or jobs for the rollout strategy, keep health validation explicit, and promote only after the staged traffic shift succeeds.

What if production deployment is approved but monitoring shows high errors afterward?

I execute rollback or forward-fix based on blast radius, use deployment history to identify the exact release, and feed the incident findings back into automated checks for future runs.

🌍 Real-World Usage

In real platform teams, multi-stage pipelines sit at the center of controlled delivery. A service is built once, containerized, pushed to ACR, validated in a lower AKS namespace, and then promoted into production only after change checks and approvals succeed.

🧾 Summary

Multi-stage pipelines turn CI into governed delivery. They let you connect build quality, artifact immutability, approvals, and environment-aware deployment into one coherent release flow.