Interview Preparation

CI (Continuous Integration) — every developer commit triggers an automated build and test cycle. The goal is to catch bugs early and provide fast feedback. If a test fails, the team knows within minutes, not days.

CD (Continuous Delivery / Continuous Deployment) — the pipeline automatically packages and delivers the application to staging or production, reducing manual errors and shipping faster.

Key distinction:

• Continuous Integration: merge → build → test (automated)
• Continuous Delivery: …→ deploy to staging (automated), deploy to production (manual gate)
• Continuous Deployment: …→ deploy to staging AND production (fully automated, no human approval)

Why it matters: Without CI/CD, teams accumulate "integration debt" — the longer you wait to merge, the more painful it becomes. CI/CD converts that big-bang integration into many small, safe, reversible changes.

Continuous Delivery: The pipeline automates everything up to staging. Production deployments require a manual approval gate — a human clicks "approve" before the code goes live. This is common in regulated industries (finance, healthcare).

Continuous Deployment: Fully automated — every commit that passes all tests goes straight to production with no human approval. This requires high confidence in your test suite and monitoring systems.

Interview tip: Most companies practice Continuous Delivery, not Deployment. Saying "we use CD with manual production gates and environment protection rules" shows pragmatism.

• Faster feedback loops — developers know within minutes if their change broke something.
• Reduced integration risk — small, frequent merges are far less risky than large, infrequent ones.
• Repeatable deployments — the same pipeline runs every time, eliminating "it works on my machine" problems.
• Audit trail — every deployment is linked to a commit, a build, and a set of test results.
• Developer confidence — teams ship more often when they trust the pipeline to catch mistakes.
• Cost reduction — catching bugs in CI is 10–100× cheaper than finding them in production.

A standard pipeline for a containerized microservice:

Lint → Unit Test → Build Docker Image → Push to Registry
  → Deploy to Staging → Integration Tests → Manual Approval
    → Deploy to Production → Smoke Test

In GitHub Actions terms:

• Lint job — ESLint, Pylint, or golangci-lint to catch style and error issues.
• Test job — run unit tests with coverage thresholds.
• Build & Push job — docker/build-push-action with image tagged as the commit SHA.
• Deploy staging job — helm upgrade --install in a staging namespace.
• Integration test job — hit staging endpoints and verify responses.
• Deploy production job — requires environment: production with reviewer approval.
• Smoke test job — verify health endpoint returns 200 in production.

Two identical environments run in parallel:

• Blue = current production (serving live traffic).
• Green = new version (deployed, tested, but not yet receiving traffic).

Once green passes health checks, the load balancer switches 100% of traffic from blue to green instantly. If something goes wrong, switch back to blue — instant rollback with zero downtime.

Trade-off: Requires double the infrastructure during the switch, so it's more expensive but safer.

Route a small percentage (5–10%) of live traffic to the new version while 90–95% continues hitting the old version. Monitor error rates, latency, and resource usage for a defined period.

• If metrics look good → gradually increase traffic to the new version (25% → 50% → 100%).
• If metrics degrade → immediately route all traffic back to the old version (rollback).

Canary deployments are less risky than blue-green because only a fraction of users are affected if something goes wrong. Often implemented via Ingress weights, service mesh (Istio), or Flagger.

Rule #1: Never run destructive migrations automatically.

Best practices:

• Run migrations as a separate step before application deployment.
• Use migration tools like Flyway, Liquibase, or language-specific tools (Alembic for Python, TypeORM for Node.js).
• Make migrations backward-compatible — the old app version should still work after the migration runs (expand-and-contract pattern).
• Separate schema migration from data migration — schema changes are fast, data backfills are slow.
• Add a manual gate before destructive operations (dropping columns, truncating tables).

jobs:
  migrate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: flyway -url=${{ secrets.DB_URL }} migrate
  deploy:
    needs: migrate    # Deploy only after migration succeeds
    runs-on: ubuntu-latest
    steps:
      - run: helm upgrade myapp ./chart

GitOps = Git as the single source of truth for both infrastructure and application state.

• CI pushes changes to a Git repository (e.g., updates an image tag in a Helm values file).
• CD is handled by a GitOps operator (ArgoCD, Flux) running inside the cluster. The operator watches Git and reconciles cluster state to match the desired state in Git.

Key difference from traditional CI/CD: the pipeline doesn't directly kubectl apply or helm upgrade. Instead, it commits to Git, and the in-cluster operator pulls and applies. This gives you: audit trail (every change is a Git commit), drift detection (operator alerts if someone changes the cluster manually), and declarative state management.

Build artifact: The compiled output of your build process — a JAR file, a Go binary, a webpack bundle, or an npm package. It's portable but needs a compatible runtime environment to execute.

Container image: An artifact + its runtime + OS-level dependencies, packaged as an OCI image (layers). It's a deployable unit — it runs identically on any machine with a container runtime.

Analogy: An artifact is the recipe. A container image is the meal-prep kit — recipe, ingredients, and utensils, all in one box. You just heat and serve.

Multiple strategies, often combined:

• Helm rollback: helm rollback myapp <previous-revision> — restores the previous Helm release.
• Re-deploy previous image: Trigger the pipeline with the previous commit SHA. Since images are tagged with SHA, this deploys the known-good version.
• Git revert: git revert <bad-commit> and push — the pipeline deploys the reverted code automatically.
• Blue-green switch: Route traffic back to the previous environment instantly.

Critical practice: Always tag images with the commit SHA, not latest. This gives you full traceability — you can always map a running container back to the exact code that produced it.

The full hierarchy:

Event (push, PR, schedule, workflow_dispatch)
  └── Workflow (.github/workflows/ci.yml)
        └── Job (runs on a specific runner)
              └── Step (a single unit: action or shell command)

• Event: The trigger — a push, a pull request, a cron schedule, or a manual dispatch.
• Workflow: A YAML file in .github/workflows/ that responds to one or more events.
• Job: A set of steps that run on the same runner. Jobs run in parallel by default, but you can create dependencies with needs:.
• Step: A single unit of work — either a shell command (run:) or an action (uses:). Steps run sequentially within a job and share the same filesystem.

Key detail: Steps share the runner workspace (filesystem + environment variables). Jobs do not — each job gets a fresh runner. To share data between jobs, use outputs or artifacts.

• uses: invokes a pre-built action — a JavaScript action, Docker action, or composite action from the Marketplace or a local path. Example: uses: actions/checkout@v4
• run: executes inline shell commands directly on the runner. Example: run: npm test

Important: You cannot combine uses: and run: in the same step. Each step is either an action invocation or a shell command, never both.

# ✅ Correct — separate steps
- uses: actions/checkout@v4
- run: npm install && npm test

# ❌ Invalid — can't mix uses and run
- uses: actions/checkout@v4
  run: npm test

Three mechanisms:

1. needs: — defines execution order. Job B waits for Job A to complete before starting.
2. Outputs — pass small data (strings) between jobs:
   yamlCopy

   jobs:
     build:
       runs-on: ubuntu-latest
       outputs:
         image-tag: ${{ steps.tag.outputs.tag }}
       steps:
         - id: tag
           run: echo "tag=sha-$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
   
     deploy:
       needs: build
       runs-on: ubuntu-latest
       steps:
         - run: echo "Deploying ${{ needs.build.outputs.image-tag }}"

3. Artifacts — share files between jobs using actions/upload-artifact and actions/download-artifact. Use for build outputs, test reports, or any data too large for outputs (which have a 1 MB limit).

GITHUB_TOKEN is an auto-generated, short-lived token created for every workflow run. Key properties:

• Scoped to the current repository — it cannot access other repos.
• Expires when the job completes — not reusable after the run.
• Permissions are configurable via the permissions: block in the workflow YAML.
• Default permissions depend on repository settings — can be "read all" or "read + write" (Settings → Actions → General → Workflow permissions).

How it differs from a PAT (Personal Access Token): PATs are created by a user, can access any repo the user has access to, and must be rotated manually. GITHUB_TOKEN is automatic, scoped, and ephemeral — always prefer it over PATs in workflows.

Choose self-hosted when: you need internal network access, custom hardware (GPUs), compliance requirements, or high volume to reduce cost.

Security warning: Never use self-hosted runners on public repositories. A malicious PR could run arbitrary code on your infrastructure.

workflow_dispatch enables manual triggering of a workflow from the GitHub UI or API, with optional custom inputs.

on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Target environment'
        required: true
        type: choice
        options: [staging, production]
      version:
        description: 'Image tag to deploy'
        required: true
        type: string

Use cases: on-demand deployments, hotfix releases, maintenance tasks (database backups, cache purges), testing workflows during development.

Matrix strategy runs the same job across multiple configurations in parallel. It creates a Cartesian product of the matrix dimensions.

strategy:
  matrix:
    node-version: [18, 20, 22]
    os: [ubuntu-latest, windows-latest]
    # Creates 3 × 2 = 6 parallel jobs

    include:
      - node-version: 22
        os: ubuntu-latest
        coverage: true      # Extra variable for this combo

    exclude:
      - node-version: 18
        os: windows-latest  # Skip this combination

Access values in steps via ${{ matrix.node-version }} and ${{ matrix.os }}. Use include to add specific combos or extra variables, and exclude to remove unwanted combos.

Rule of thumb: If you need full jobs with environments and secrets → reusable workflow. If you need a few steps bundled together → composite action.

The concurrency key groups workflow runs and controls what happens when multiple runs of the same group are active.

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

How it works: If a new run starts with the same concurrency group, the previous in-progress run is cancelled. This is critical for PR builds — if a developer pushes 3 commits in quick succession, only the latest commit's run proceeds.

Without cancel-in-progress: the new run queues until the previous one finishes.

needs: creates a Directed Acyclic Graph (DAG) of job execution order.

jobs:
  lint:     ...          # Runs immediately
  test:     ...          # Runs immediately (parallel with lint)
  build:
    needs: [lint, test]  # Waits for BOTH lint and test
  deploy:
    needs: build         # Waits for build

Behavior when a needed job fails: all dependent jobs are skipped. To override this, use if: always() or if: failure() on the dependent job.

Accessing outputs: ${{ needs.build.outputs.image-tag }} — you can only access outputs from jobs listed in your needs:.

name: CI Pipeline              # Display name in the Actions tab

on:                             # Trigger events
  push:
    branches: [main]
  pull_request:
    branches: [main]

permissions:                    # Token permissions (least privilege)
  contents: read

env:                            # Workflow-level environment variables
  REGISTRY: myacr.azurecr.io

jobs:
  build:                        # Job ID
    runs-on: ubuntu-latest      # Runner selection
    needs: [lint]               # Job dependency (optional)
    if: github.ref == 'refs/heads/main'  # Condition (optional)
    strategy:                   # Matrix (optional)
      matrix:
        node: [18, 20]
    steps:
      - uses: actions/checkout@v4        # Action step
      - run: npm install                 # Shell step
        env:
          NODE_ENV: production           # Step-level env var

Order matters for readability, but only on: and jobs: are required at the top level.

• Tabs instead of spaces — YAML only allows spaces. A single tab will silently break the workflow.
• Inconsistent indentation — mixing 2-space and 4-space indentation within the same level.
• Unquoted colons in run: — run: echo Time: 12:30 breaks because YAML interprets the second colon as a key-value separator.
• Missing ${{ }} in expressions — outside if:, you must wrap expressions: ${{ env.MY_VAR }}. Inside if:, expressions are implicit.
• Duplicate keys — two env: blocks at the same level silently overwrite.
• Wrong file path — workflow files must be in .github/workflows/ with a .yml or .yaml extension.

on:
  push:
    branches: [main, 'release/**']     # Only these branches
    paths: ['src/**', 'package.json']   # Only when these files change
    paths-ignore: ['docs/**', '*.md']   # Ignore these files

Rules:

• branches and paths can be combined — both must match.
• paths and paths-ignore cannot be used together.
• Path filters use glob patterns: * matches single level, ** matches multiple levels.
• If only docs changed and your paths filter doesn't include docs, the workflow won't run.

Conditional execution at the job or step level:

# Only deploy from main branch
- run: helm upgrade myapp ./chart
  if: github.ref == 'refs/heads/main'

# Only run on pull requests
- run: echo "This is a PR"
  if: github.event_name == 'pull_request'

# Run even if previous steps failed
- run: echo "Sending failure notification"
  if: failure()

# Always run (cleanup)
- run: rm -rf ./tmp
  if: always()

# Skip CI based on commit message
- run: npm test
  if: "!contains(github.event.head_commit.message, '[skip ci]')"

Note: Expressions in if: are automatically wrapped in ${{ }}, so you don't need to write them explicitly.

The expression evaluator for GitHub Actions. It can access contexts and call functions:

Contexts: github, env, secrets, matrix, needs, steps, runner, job, inputs

Functions:

• contains(search, item) — string or array contains check.
• startsWith(string, prefix) / endsWith()
• toJSON(value) / fromJSON(value) — serialize/deserialize (critical for dynamic matrices).
• hashFiles('**/package-lock.json') — generates a hash for cache keys.
• format() — string formatting.

Key rule: Inside if:, expressions are implicit. Everywhere else, you must use ${{ }}.

Between steps (same job):

steps:
  - id: version
    run: echo "tag=v1.2.3" >> $GITHUB_OUTPUT
  - run: echo "Version is ${{ steps.version.outputs.tag }}"

Between jobs:

jobs:
  build:
    outputs:
      tag: ${{ steps.version.outputs.tag }}
    steps:
      - id: version
        run: echo "tag=v1.2.3" >> $GITHUB_OUTPUT
  deploy:
    needs: build
    steps:
      - run: echo "${{ needs.build.outputs.tag }}"

Important: $GITHUB_OUTPUT replaced the deprecated set-output command. Always use the file-based approach.

• continue-on-error: true — the step/job reports success even if it fails. The overall job status stays green. Use for non-critical steps (e.g., optional code quality tools).
• if: always() — the step runs regardless of previous step results, but its own result affects the job status normally. Use for cleanup or notification steps.

Example: A notification step should use if: always() so it runs when things fail. An experimental lint tool should use continue-on-error: true so it doesn't block the pipeline.

Default job timeout is 360 minutes (6 hours). This is dangerously high for most workflows.

jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 15       # Job-level timeout
    steps:
      - run: npm test
        timeout-minutes: 10   # Step-level timeout

Best practice: Always set a timeout. A stuck npm install or a test that hangs can run for 6 hours, consuming your minutes budget. Set to 2–3× the expected duration.

Three scopes, from broadest to narrowest:

1. Organization secrets — shared across selected (or all) repositories in the org.
2. Repository secrets — available to all workflows in that repo.
3. Environment secrets — only available to jobs that reference that specific environment.

Override order: Environment > Repository > Organization. If the same secret name exists at multiple levels, the most specific scope wins.

Security properties:

• Encrypted at rest using libsodium sealed boxes.
• Automatically masked in logs (the raw value is replaced with ***).
• Not accessible in fork PRs — this prevents secret exfiltration from malicious forks.
• Never exposed in workflow YAML — accessed only through ${{ secrets.NAME }}.

OIDC (OpenID Connect) enables passwordless authentication between GitHub Actions and cloud providers (Azure, AWS, GCP).

How it works:

1. GitHub Actions presents a JWT (JSON Web Token) with claims about the workflow (repo, branch, environment).
2. The cloud provider validates the JWT against GitHub's OIDC endpoint.
3. If the claims match the trust policy, the provider issues short-lived credentials (15–60 minutes).

Why it's better than stored secrets:

• No secret rotation — there's nothing stored to rotate.
• No leak risk — no long-lived credentials exist anywhere.
• Fine-grained trust — the cloud provider can restrict access by repo, branch, environment, and even workflow.
• Audit trail — every token request is logged with full context.

permissions:
  id-token: write   # Required for OIDC
  contents: read

steps:
  - uses: azure/login@v2
    with:
      client-id: ${{ secrets.AZURE_CLIENT_ID }}
      tenant-id: ${{ secrets.AZURE_TENANT_ID }}
      subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

Trade-off: OIDC is better but requires initial setup in Azure AD (federated credentials). Worth the one-time effort for every production workflow.

Common causes:

• Secret used in a URL: git clone https://user:${{ secrets.TOKEN }}@github.com/... — the URL gets logged and the masking may not catch it as part of a larger string.
• Base64 encoded: masking only hides the original value. The encoded version is a different string.
• Sub-string match failure: very short secrets (e.g., 3 characters) might not get masked reliably.
• Environment variable expansion: some tools echo environment variables as part of their verbose output.

Fixes:

• Always pass secrets via env: blocks, never inline in run:.
• Use add-mask for derived values: echo "::add-mask::$DERIVED_VALUE"
• Suppress output for sensitive commands: redirect to file or /dev/null.
• Audit all steps that reference secrets and check their output behavior.

Third-party actions run code on your runner with access to your secrets. Best practices:

• Pin by SHA, not tag: uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 — tags can be moved; SHAs are immutable.
• Enable Dependabot for actions: automatically creates PRs when pinned actions have updates.
• Review source code before first use — check what the action does with secrets and network access.
• Limit to trusted publishers — GitHub verified creators badge.
• Use org-level allow-lists — restrict which actions can be used across all repositories.

Follow the principle of least privilege:

permissions:
  contents: read       # Read repo code
  id-token: write      # OIDC authentication
  # Everything else: implicitly denied

Additionally:

• Environment protection rules — required reviewers for the production environment.
• Wait timers — enforce a delay between staging and production deploys.
• Branch restrictions — only allow deployments from main or release/* branches.
• Deployment branch policy — prevent ad-hoc deployments from feature branches.

Never use permissions: write-all in production workflows.

When a fork opens a PR against your repo:

• GITHUB_TOKEN is read-only — cannot push, create releases, or modify the target repo.
• Secrets are NOT available — ${{ secrets.MY_SECRET }} resolves to an empty string.
• First-time contributors require maintainer approval before any workflow run (configurable in Settings → Actions → Fork pull request workflows).

Why? This prevents secret exfiltration. A malicious actor could fork your repo, modify the workflow to echo ${{ secrets.PROD_PASSWORD }}, and steal credentials. The fork restriction prevents this entirely.

1. Create environments in Settings → Environments (e.g., staging, production).
2. Add protection rules:
   • Required reviewers — 1+ people must approve before the job proceeds.
   • Wait timer — enforce a cooldown (e.g., 30 minutes between staging and production).
   • Branch restrictions — only main can deploy to production.
3. Reference in workflow:

jobs:
  deploy-production:
    runs-on: ubuntu-latest
    environment: production      # ← This pauses for approval
    steps:
      - run: helm upgrade myapp ./chart --namespace production

The job will pause and wait for a reviewer to approve in the GitHub UI before executing any steps.

• GitHub audit log — tracks which workflows accessed which secrets and when.
• Code search — search all workflow files for secrets. references: org:myorg path:.github/workflows "secrets."
• Org-level secret policies — centrally manage which repos can access which secrets.
• Migrate to OIDC — reduces the number of stored secrets that need auditing.
• Automate with a workflow — create a scheduled workflow that scans all repos' workflow files and reports secret usage.

Full pipeline architecture:

┌──────┐   ┌──────┐   ┌────────────┐   ┌──────────┐
│ Lint │──▶│ Test │──▶│ Build+Push │──▶│ Deploy   │
└──────┘   └──────┘   │ Docker→ACR │   │ Staging  │
                       └────────────┘   └────┬─────┘
                                             │
                                        ┌────▼─────┐   ┌──────────┐
                                        │ Smoke    │──▶│ Approval │
                                        │ Test     │   │ Gate     │
                                        └──────────┘   └────┬─────┘
                                                             │
                                                        ┌────▼──────┐
                                                        │ Deploy    │
                                                        │ Production│
                                                        └───────────┘

name: Deploy to AKS
on:
  push:
    branches: [main]

permissions:
  id-token: write
  contents: read

env:
  ACR_NAME: myacr
  IMAGE: myacr.azurecr.io/myapp

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm run lint

  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm test

  build-push:
    needs: [lint, test]
    runs-on: ubuntu-latest
    outputs:
      image-tag: ${{ steps.meta.outputs.tag }}
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - run: az acr login --name $ACR_NAME
      - id: meta
        run: echo "tag=sha-$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
      - run: |
          docker build -t $IMAGE:${{ steps.meta.outputs.tag }} .
          docker push $IMAGE:${{ steps.meta.outputs.tag }}

  deploy-staging:
    needs: build-push
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - run: az aks get-credentials --resource-group myRG --name myAKS
      - run: |
          helm upgrade --install myapp ./chart \
            --namespace staging \
            --set image.tag=${{ needs.build-push.outputs.image-tag }} \
            --atomic --wait --timeout 5m

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production    # Pauses for approval
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - run: az aks get-credentials --resource-group myRG --name myAKS
      - run: |
          helm upgrade --install myapp ./chart \
            --namespace production \
            --set image.tag=${{ needs.build-push.outputs.image-tag }} \
            --atomic --wait --timeout 5m

Key decisions: OIDC for Azure auth (no stored credentials), --atomic --wait for safe deploys (auto-rollback on failure), commit SHA tags for traceability, environments for approval gates.

Step-by-step diagnosis:

1. Check GitHub Actions logs — identify which step failed. Is it the helm upgrade step? What does the error say?
2. Check pod status:
   bashCopy

   kubectl get pods -n production
   # Look for: CrashLoopBackOff, ImagePullBackOff, Pending

3. Describe the failing pod:
   bashCopy

   kubectl describe pod <pod-name> -n production
   # Check Events section: image pull errors, resource limits, scheduling failures

4. Check application logs:
   bashCopy

   kubectl logs <pod-name> -n production
   kubectl logs <pod-name> -n production --previous  # Previous crash

5. Verify the image exists in ACR: az acr repository show-tags --name myacr --repository myapp
6. Check resource quotas: kubectl describe resourcequota -n production
7. Check readiness/liveness probes — misconfigured probes are the #1 cause of Helm timeouts.
8. Rollback if needed: helm rollback myapp -n production

Root cause hierarchy: 80% of Helm timeouts are caused by: 1) bad readiness probes, 2) wrong image tag, 3) insufficient resources, 4) missing secrets/configmaps.

1. Deploy the canary as a separate Helm release with 1 replica and a low traffic weight:
   bashCopy

   helm upgrade --install myapp-canary ./chart \
     --set replicaCount=1 \
     --set ingress.canaryWeight=10 \
     --namespace production

2. Run health checks for N minutes — monitor error rates and latency via Prometheus/Grafana or an API health endpoint.
3. If healthy → promote: update the main release with the new image and remove the canary.
   bashCopy

   helm upgrade myapp ./chart --set image.tag=$NEW_TAG
   helm uninstall myapp-canary

4. If unhealthy → rollback: remove the canary, main release is untouched.
   bashCopy

   helm uninstall myapp-canary

For advanced canary, use Flagger with Istio or NGINX — it automates the traffic shifting and analysis.

Automatic rollback: Use --atomic flag with helm upgrade. If the deployment fails (pods don't become ready within the timeout), Helm automatically rolls back to the previous release.

helm upgrade --install myapp ./chart --atomic --wait --timeout 5m

Manual rollback:

helm history myapp -n production          # List revisions
helm rollback myapp 3 -n production       # Rollback to revision 3

CI-triggered rollback: Create a separate workflow_dispatch workflow that accepts a revision number and runs helm rollback. Or simply re-run a previous successful workflow — since images are tagged with commit SHA, it deploys the known-good version.

Best practice: Always test your rollback procedure. A rollback you've never tested is a rollback that will fail when you need it most.

Optimization strategies (ordered by impact):

1. Parallelize independent jobs — lint, test, and security scan can run in parallel instead of sequentially.
2. Add dependency caching:
   yamlCopy

   - uses: actions/setup-node@v4
     with:
       node-version: 20
       cache: 'npm'    # Built-in caching

3. Docker layer caching — use cache-from: type=gha with docker/build-push-action.
4. Reduce test scope for PRs — only run tests for changed modules using path filters or a test impact analysis tool.
5. Use larger runners — ubuntu-latest-4-cores (GitHub-hosted) for compute-heavy jobs.
6. Skip unnecessary steps — use path filters to skip CI for docs-only changes.
7. Cancel stale runs — use concurrency with cancel-in-progress: true.

Key strategies:

• Path filters per service: each workflow triggers only when its service's files change.
• Dynamic matrix to detect changed services and build only those:
  yamlCopy

  - uses: dorny/paths-filter@v3
    id: changes
    with:
      filters: |
        api: 'services/api/**'
        web: 'services/web/**'
        auth: 'services/auth/**'
  
  # Later: build a dynamic matrix from the changed services
  - id: matrix
    run: |
      echo "services=$(echo '${{ toJSON(steps.changes.outputs) }}' | jq -c '[to_entries[] | select(.value=="true") | .key]')" >> $GITHUB_OUTPUT

• Reusable workflows for standardized CI — every service calls the same build-and-deploy workflow.
• Per-service Helm charts — each service has its own chart in services/<name>/chart/.
• Parallel deploys — use a matrix job to deploy all changed services simultaneously.

Strategy:

1. Deploy to primary region first.
2. Run smoke tests against the primary region.
3. Deploy to secondary regions in parallel using a matrix strategy:
   yamlCopy

   deploy-secondary:
     needs: deploy-primary
     strategy:
       matrix:
         region: [westeurope, southeastasia, brazilsouth]
         include:
           - region: westeurope
             cluster: aks-eu
             rg: rg-eu
           - region: southeastasia
             cluster: aks-asia
             rg: rg-asia
           - region: brazilsouth
             cluster: aks-brazil
             rg: rg-brazil

4. Use environment-per-region with individual approval gates if needed.

Considerations: DNS failover (Azure Traffic Manager / Front Door), database replication lag (deploy DB migrations to primary first, wait for replication), and regional configuration differences (CDN endpoints, compliance settings).

Common causes:

• Secrets not available in fork: fork PRs don't receive secrets from the upstream repo. The workflow "passes" in the fork because the step that needs the secret silently uses an empty string.
• Different branch protection rules: the main repo may require status checks or CODEOWNERS review that the fork doesn't.
• Org policy restrictions: the main repo's organization may restrict which actions are allowed.
• Environment protection rules: environments with required reviewers exist in the main repo but not the fork.
• Different runner availability: the main repo might use org-specific self-hosted runners that the fork doesn't have access to.
• CODEOWNERS restrictions: changes to certain paths require review from specific code owners in the main repo.

Debugging approach: Compare the workflow runs — check exactly which step fails and what the error message says. Most often it's a secrets or permissions issue.

Phased approach (don't try to do everything at once):

Key principles:

• Start small and iterate — don't build the entire pipeline in week 1.
• Make it the default — remove the old SSH workflow completely once the pipeline is stable.
• Get developer buy-in — show them the pipeline saves time, don't force it.
• Document everything — the team needs to understand and maintain the pipeline without you.

• Central .github repository — contains reusable workflows that all repos call via workflow_call. Update once, all 200 repos benefit.
• Org-level secrets and variables — shared credentials (registry URL, OIDC config) managed centrally.
• Custom actions for common patterns — create composite actions for standard tasks (lint, build Docker, deploy Helm).
• Required status checks — branch protection rules enforce that specific workflows must pass before merging.
• Org-wide runner groups — self-hosted runners shared across repos with proper access controls.
• Dependabot for action updates — a single Dependabot config template keeps all repos' action versions current.
• Internal documentation — "How to CI/CD at [Company]" guide with templates and examples.

Incident response sequence:

1. Verify the pipeline — check GitHub Actions logs. All steps green? Build, push, and deploy all succeeded?
2. Check pod status:
   bashCopy

   kubectl get pods -n production
   # Are pods Running? Ready? Restarting?

3. Check application logs: kubectl logs <pod> -n production — look for runtime errors, connection failures, OOM kills.
4. Check ingress / DNS: Is the ingress controller healthy? Did the DNS update propagate? Can you curl the health endpoint directly from inside the cluster?
5. Check health endpoints: curl -v https://myapp.com/healthz
6. Compare Helm release values with expected: helm get values myapp -n production
7. Rollback immediately: helm rollback myapp -n production — restore service first, investigate later.
8. Post-mortem: Add smoke tests to the pipeline (they would have caught this), add readiness probe checks, add synthetic monitoring.

Golden rule: Restore service first, investigate second. A 5-minute rollback is better than 30 minutes of debugging while users are impacted.

Core principle: Decouple deployment from release.

• Deployment = code is on production servers (handled by CI/CD).
• Release = feature is visible to users (handled by feature flags).

Implementation:

1. Use a feature flag service — LaunchDarkly, Unleash (open source), or a simple config-based system.
2. Deploy code with flags off — the feature is in production but invisible to users.
3. Enable gradually via the flag service — 5% → 25% → 50% → 100%.
4. If issues arise, disable the flag instantly — no redeployment needed.

CI/CD integration: The pipeline deploys code. The flag service controls behavior. They are independent systems. Never use CI/CD to toggle feature flags — that defeats the purpose of instant rollback.

Compliance-focused pipeline requirements:

• Signed commits — require GPG or SSH commit signatures via branch protection rules.
• Required reviewers (2+) — enforce four-eyes principle for all production changes.
• SOC2-compliant audit trail — GitHub's audit log captures every workflow run, every secret access, every deployment event.
• OIDC only — no stored secrets for cloud authentication. Eliminates credential rotation compliance burden.
• SHA-pinned actions — immutable action references prevent supply chain attacks.
• SLSA provenance — generate build provenance attestations (actions/attest-build-provenance) to prove what code produced which artifact.
• Environment wait timers — mandatory cooling periods between staging and production.
• Separate environments — staging → pre-production → production, each with its own approval chain.
• Automated compliance scanning — add SAST (CodeQL), dependency scanning (Dependabot), container scanning (Trivy) as required pipeline stages.

Cost reduction strategies (ordered by impact):

1. Analyze runner minutes usage — find the top 10 workflows by minutes consumed. Focus optimization efforts there.
2. Add path filters — skip CI entirely for docs-only, README, or config-only changes.
3. Enable caching — npm ci with cache saves 1–3 minutes per run. Docker layer caching saves 5+ minutes.
4. Use concurrency — cancel stale runs. If a developer pushes 5 commits, only the last one needs to complete.
5. Reduce matrix scope for PRs — run the full matrix (all OS × all versions) only on main. For PRs, test on the most common configuration only.
6. Consider self-hosted runners — for high-volume repos (100+ runs/day), the break-even point is often reached within a month.
7. Use spot instances for non-critical jobs — self-hosted runners on spot/preemptible VMs cost 60–90% less.
8. Right-size runner specs — not every job needs a 4-core runner. Use the default 2-core for simple tasks.

Quick win: Adding concurrency: cancel-in-progress and path filters alone often reduces costs by 30–40%.