Troubleshooting

Debug the most common Prometheus failures: metrics not collected, targets down, empty graphs, noisy alerts, and broken Kubernetes scraping.

Simple Explanation (ELI5)

Troubleshooting Prometheus usually means answering one question first: did Prometheus fail to discover the target, fail to reach it, fail to store it, or did we just ask the wrong query?

Real-world Analogy

If a parcel never arrives, you check the address, then the delivery route, then the warehouse, then the recipient. Prometheus troubleshooting works the same way: target, network, scrape, storage, then query.

Technical Explanation

The fastest troubleshooting path is layered. Start with target discovery, then endpoint reachability, then scrape status, then raw metric presence, then PromQL and dashboards. In Kubernetes, also inspect ServiceMonitor or PodMonitor selectors, namespace visibility, and network policies.

Visual Representation

Commands / Syntax

# Prometheus APIs
curl http://localhost:9090/api/v1/targets
curl http://localhost:9090/api/v1/label/__name__/values
curl http://localhost:9090/api/v1/query?query=up

# Check endpoint directly
curl http://my-app:8080/metrics | head

# Kubernetes checks
kubectl get servicemonitor,podmonitor -A
kubectl describe servicemonitor checkout-api -n monitoring
kubectl get svc,pods -n prod -l app=checkout-api -o wide
kubectl logs deploy/monitoring-kube-prometheus-operator -n monitoring

Example (Real-world Use Case)

A new service in production has no dashboard data. The operator logs show no scrape config generated for the ServiceMonitor. Investigation reveals the ServiceMonitor lives in a namespace not watched by the Prometheus instance. Fixing the namespace selector immediately restores metrics.

Hands-on Section

1. Break one target intentionally by changing its scrape port.
2. Watch the target switch to DOWN in the Prometheus UI.
3. Restore connectivity and verify up returns to 1.
4. Break a ServiceMonitor selector and observe how the target disappears entirely.

Try It Yourself

• Write a step-by-step checklist for “metrics not collected.”
• Explain the difference between “target is down” and “target is not discovered.”
• Name one API endpoint that helps inspect raw Prometheus target state.

Debugging Runbook 1: Metrics Not Collected

• Check whether the target exists in Status → Targets.
• If absent, investigate discovery, selectors, annotations, ServiceMonitor, or PodMonitor.
• If present but down, test endpoint reachability and path.
• If present and up, query the raw metric name before touching Grafana.

Debugging Runbook 2: Kubernetes ServiceMonitor Not Working

• Compare Service labels to ServiceMonitor selector exactly.
• Verify the service port name matches the monitor endpoint port name.
• Check Prometheus namespace selectors and RBAC scope.
• Review operator logs for rejected or ignored monitor resources.

Debugging Runbook 3: Empty Dashboard but Healthy Target

• Check whether the dashboard query uses the correct metric and labels.
• Query the raw metric in Prometheus directly.
• Verify the time range includes recent data.
• Check whether recording rules failed or changed metric names.

Debugging Runbook 4: Alert Fires with No Real Incident

• Check whether the alert expression is too sensitive.
• Inspect whether the for clause is missing or too short.
• Correlate with user-impact metrics like latency and error rate.
• Review planned maintenance, rollout windows, or autoscaler behavior.

Interview Questions

Summary

Prometheus troubleshooting is most effective when you stay disciplined: discovery first, connectivity second, raw metric third, PromQL fourth, dashboards last. In Kubernetes, selectors, port names, namespaces, and RBAC are recurring failure points.