Introduction to Prometheus
Learn what Prometheus is, why it became the default cloud-native monitoring system, and how its pull model differs from older monitoring stacks.
Simple Explanation (ELI5)
Prometheus is a system that regularly visits your apps, asks for their numbers, stores those numbers, and lets you search them later. Instead of waiting for apps to push data somewhere, Prometheus pulls metrics from endpoints like /metrics.
Real-world Analogy
Imagine a warehouse manager walking the floor every minute with a clipboard, checking stock, temperature, and machine status. That manager is Prometheus. Each machine that exposes its numbers is an exporter or instrumented app.
Technical Explanation
Prometheus is an open-source monitoring system and time-series database. It stores labeled time-series data, scrapes targets over HTTP, evaluates rules, and can send alerts to Alertmanager. It was built for dynamic environments like Kubernetes, where instances appear and disappear frequently.
| Capability | Prometheus Behavior | Why It Matters |
|---|---|---|
| Collection model | Pull-based scraping | Simple, auditable, target health is visible |
| Storage | Local TSDB | Fast reads for dashboards and alerts |
| Query language | PromQL | Rich aggregation and alert logic |
| Discovery | Static and service discovery | Works in dynamic platforms like Kubernetes |
| Alerting | Rules + Alertmanager | Decouples signal generation from notification routing |
Visual Representation
Apps
Exporters
Scrape + Store
Alertmanager
Commands / Syntax
global:
scrape_interval: 15s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]# Run Prometheus locally with Docker docker run --name prometheus -p 9090:9090 \ -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml \ prom/prometheus # Open UI curl http://localhost:9090/-/ready curl http://localhost:9090/api/v1/targets
Example (Real-world Use Case)
A platform team deploys Prometheus into a Kubernetes cluster to monitor API pods, ingress latency, node resources, kube-state-metrics, and application error rates. Prometheus becomes the data source for Grafana dashboards and SRE alerting.
Hands-on Section
- Create a minimal
prometheus.ymlwith one static target. - Run Prometheus in Docker and verify the UI on port 9090.
- Open the
Status → Targetspage and confirm the target is up. - Run the expression
upin the UI.
Try It Yourself
- Change the target port to a non-existent one and observe how Prometheus shows target failure.
- Describe one advantage of pull-based scraping over push-based delivery.
- Find the difference between Prometheus and Grafana in one sentence.
Debugging Scenarios
Prometheus not collecting data is often not a storage problem. It is usually target reachability, bad discovery labels, or an incorrectly exposed /metrics endpoint.
- If the target shows
DOWN, inspect endpoint accessibility and the target address. - If the UI is up but graphs are empty, check the query range and whether the scrape target actually exports metrics.
- If Kubernetes integration is expected, verify RBAC and service discovery configuration rather than editing random queries first.
Interview Questions
Beginner
Prometheus is an open-source monitoring system and time-series database used to collect, store, query, and alert on metrics.
It stores labeled time-series data, where each metric sample has a timestamp, value, and set of labels.
Prometheus uses a pull model. It scrapes metrics from targets over HTTP.
PromQL is Prometheus Query Language, used to select, transform, and aggregate metrics.
Alertmanager receives alerts from Prometheus and handles grouping, deduplication, silencing, and routing to channels like Slack or email.
Intermediate
Prometheus can discover short-lived targets dynamically and show whether a target is reachable, which is useful in constantly changing clusters.
Short-lived batch jobs can push final metrics to the Pushgateway, but Prometheus itself is still fundamentally scrape-oriented.
Prometheus is great at collection, querying, and rule evaluation, while Grafana provides more flexible dashboards and visualization options.
Exporters translate metrics from systems that do not natively expose Prometheus format, such as Linux hosts, MySQL, or Redis.
A scrape target is an endpoint that exposes metrics in Prometheus text format, usually over HTTP.
Scenario-based
Generally no. Prometheus is designed to scrape. For short-lived jobs, use Pushgateway carefully. For long-running services, expose a /metrics endpoint instead.
Check target health in Prometheus first, then verify the Grafana data source, query range, and metric names. Do not start with Grafana styling.
Start with node exporter and a few high-value service metrics, show dashboards and alerting wins, then gradually replace ad hoc scripts with reproducible metric collection.
Because Prometheus supports service discovery and can track dynamic targets using Kubernetes metadata rather than fixed static IPs.
I check whether Prometheus can reach the correct address and port, whether the path is correct, and whether labels or discovery are targeting the intended pod or service.
Summary
Prometheus became the standard for cloud-native monitoring because it combines pull-based collection, flexible querying, strong service discovery, and straightforward alerting. The next step is learning the metric data model that makes this possible.