intermediate
Product analytics and experimentation
Connect feature work to event design, funnels, guardrail metrics, experiment validity, rollout decisions, and privacy-aware measurement.
Product analytics and experimentation connect engineering work to **measurable outcomes**: event instrumentation, funnels, cohorts, guardrail metrics, A/B tests, and rollout decisions. Interviews test whether you can design trustworthy measurement—not only ship a feature flag.
| Building block | Purpose | |----------------|---------| | Event schema | Stable names, required properties, versioning, idempotency keys | | Identity | Anonymous vs logged-in stitching without violating privacy | | Funnels & retention | Step definitions that match real UX, not page URLs only | | Guardrails | Revenue, latency, errors, support volume alongside primary metric | | Experiment design | Hypothesis, assignment unit, duration, power, stopping rules | | Rollout | Gradual exposure, kill switches, holdouts for long-term effects |
Before launch: primary metric + guardrails + assignment logic + privacy review
After launch: sanity checks (SRM, sample ratio), then decision memo
Validity traps: peeking early; changing metrics mid-flight; overlapping experiments on the same users; novelty effects; seasonality; measuring clicks instead of completed outcomes.
Privacy-aware measurement: minimize properties, honor consent, aggregate where possible, and document data flows to analytics vendors.
On interviews: prove a new onboarding helps activation. Define events, primary metric (e.g., day-7 activation), guardrails (crash rate, time-to-first-value), assignment (user-level), and how you would detect a broken experiment.
Common pitfalls: tracking page views only; no control group; shipping to 100% before instrumentation works; analytics that cannot be reproduced from warehouse SQL.
The trade-off is measurement rigor and experiment latency versus speed of shipping and organizational appetite for inconclusive results.
Checklist:
- Event dictionary reviewed with PM and data.
- Pre-registered primary and guardrail metrics.
- Assignment and exposure logging verifiable.
- Rollback path if guardrails breach.