advanced
Distributed tracing
Follow a request through frontend, gateway, service, database, and worker hops to locate latency and failure boundaries.
Distributed tracing follows one logical operation across processes: browser → API → database → queue → worker → third-party HTTP. It exposes where latency accumulates and which dependency owns a failure.
[browser]──►[api]──►[postgres]
└──►[payment-api]
└──►[queue]──►[worker]
Sampling balances cost and coverage: always trace errors or slow paths; sample happy paths. Traces become useful when they share IDs with logs and metrics exemplars.
On interviews: critical path for checkout, head-based vs tail-based sampling, and tracing async work that continues after the HTTP response.
Common pitfalls: broken context at queue boundaries; tracing only the frontend; spans too granular (every function) or too coarse (one span per service).
The trade-off is storage and instrumentation overhead versus mean time to understand cross-service incidents.
Checklist:
- Propagate context on HTTP and messaging.
- Create spans at dependency boundaries.
- Link traces to logs via trace ID.
- Sample strategically, not uniformly 100%.