advanced

Prometheus

Collect and query metrics with pull scraping, labels, PromQL, alert rules, and careful cardinality control.

Prometheus scrapes metrics from exporters and instrumented apps, stores local TSDB blocks, and evaluates alerting rules. It pairs naturally with Grafana and Alertmanager for notification routing.

					scrape_configs:
  - job_name: api
    metrics_path: /metrics
    static_configs:
      - targets: ['api:9090']

| Component | Role | |-----------|------| | Exporters | node, postgres, blackbox probes | | Alertmanager | Group, inhibit, route pages | | Service discovery | Kubernetes pod targets |

Run Prometheus with retention aligned to incident lookback. Federation or remote write scales long-term storage when needed.

On interviews: HA patterns, cardinality incidents, and blackbox vs whitebox monitoring.

Common pitfalls: scraping through load balancers without consistent labels; alert storms without inhibition; running without resource limits.

The trade-off is operational simplicity in-cluster versus long-term storage and HA complexity.

Checklist:

Discover targets dynamically in K8s.
Route alerts through Alertmanager.
Monitor Prometheus itself.
Plan retention and remote write early.