Integration (Kubernetes, Apps)

Integrate Prometheus with Kubernetes and application instrumentation. This is the most important production integration path for modern teams.

Simple Explanation (ELI5)

Prometheus becomes powerful when it can automatically discover Kubernetes workloads and scrape app metrics without you manually updating IP addresses every day.

Real-world Analogy

Kubernetes is like a busy airport where gates, crews, and planes change constantly. Prometheus needs a live flight board, not a printed list from yesterday. That flight board is service discovery and the Prometheus Operator.

Technical Explanation

In Kubernetes, Prometheus is commonly deployed with the Prometheus Operator. The operator introduces custom resources like ServiceMonitor, PodMonitor, and PrometheusRule. These resources define what to scrape and how. Core integrations usually include node-exporter, kube-state-metrics, cAdvisor-derived container metrics, kubelet metrics, and application metrics exposed by pods or services.

Visual Representation

Commands / Syntax

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout-api
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: checkout-api
  namespaceSelector:
    matchNames:
      - prod
  endpoints:
    - port: metrics
      path: /metrics
      interval: 15s

# Install kube-prometheus-stack with Helm
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

# Check core components
kubectl get pods -n monitoring
kubectl get servicemonitors -A
kubectl get podmonitors -A
kubectl port-forward svc/monitoring-kube-prometheus-prometheus -n monitoring 9090:9090

from prometheus_client import Counter, Histogram, start_http_server

requests_total = Counter("checkout_requests_total", "Total checkout requests", ["status"])
request_latency = Histogram("checkout_request_duration_seconds", "Checkout latency")

start_http_server(8000)

# In app code:
requests_total.labels(status="200").inc()
with request_latency.time():
    process_checkout()

Example (Real-world Use Case)

A retail platform runs kube-prometheus-stack in the monitoring namespace. ServiceMonitor resources discover API services by label. kube-state-metrics exposes deployment readiness, node exporter exposes host pressure, and app instrumentation exposes business endpoints like checkout and payment metrics.

Hands-on Section

1. Install kube-prometheus-stack in a test cluster.
2. Create a service with a metrics named port.
3. Add a matching ServiceMonitor in the monitoring namespace.
4. Confirm the target appears in Prometheus and query up{job=~".*checkout.*"}.

Try It Yourself

• Explain when you would choose PodMonitor over ServiceMonitor.
• List the minimum metrics you want from a Kubernetes node and a pod.
• Instrument one demo app endpoint with a counter and histogram.

Debugging Scenarios

• If ServiceMonitor exists but target is absent, compare selector labels to the Service labels exactly.
• If target exists but is down, verify path, scheme, port name, and pod connectivity.
• If kube-state-metrics is missing, check that the deployment is running and being scraped by the operator-managed Prometheus instance.

Interview Questions

Summary

Kubernetes integration is where Prometheus becomes operationally decisive. The combination of Operator resources, node and cluster exporters, and application instrumentation gives teams both platform-level and user-level visibility.