Prometheus basics

Prometheus pulls metrics from targets on a schedule, stores time series keyed by metric name and labels, and evaluates PromQL for dashboards and alerts. It is pull-based, label-centric, and common in Kubernetes stacks.

| Type | Use | |------|-----| | Counter | Monotonically increasing (requests, errors) | | Gauge | Point-in-time value (queue depth, memory) | | Histogram | Latency/count distributions with buckets |

					sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))
				

Cardinality explodes when labels include user IDs or unbounded paths—design label sets deliberately. Recording rules and alert windows (`for: 5m`) reduce flapping.

On interviews: pull vs push, histogram buckets, exemplars linking to traces, and alert rule design.

Common pitfalls: `rate()` on gauges; too many histogram buckets; alerting on raw counters without aggregation.

The trade-off is powerful ad-hoc querying versus operational cost of label discipline.

Checklist:

• Prefer counters + histograms for RED.
• Bound label dimensions.
• Use recording rules for heavy queries.
• Test alert rules against historical data.