Benchmark pitfalls

Benchmarks lie easily. Common traps: cold JVM/Node warmup ignored, unrealistic data, coordinated omission, caching artifacts, comparing averages only, noisy shared CI machines, and micro-benchmarks that do not reflect system behavior.

| Pitfall | Effect | |---------|--------| | No warmup | First runs skew results | | Tiny dataset | Everything fits L1 cache | | Coordinated omission | Tool waits for slow responses, hides tail latency | | Average only | p99 disaster invisible | | Optimizing the loop | V8 dead-code elimination on unused results |

					// Benchmark hygiene: warmup + multiple iterations
for (let i = 0; i < 50; i++) run();
const samples = [];
for (let i = 0; i < 200; i++) samples.push(time(run));
reportPercentiles(samples);
				

Prefer **production-like** workloads and A/B deploys with field metrics over localhost micro-benchmark wins. When using `benchmark.js` or similar, prevent optimization from deleting work.

On interviews: coordinated omission explained; why p95/p99 matter; regression in lab but not field (or reverse); statistical significance with variance.

Common pitfalls: declaring victory from one Lighthouse run; benchmarking dev mode bundles; comparing different hardware without normalization.

The trade-off is fast local feedback versus trustworthy conclusions that survive production.

Checklist:

• Warm up; report percentiles.
• Use realistic data sizes.
• Cross-check with traces and RUM.
• Skeptically review micro-benchmark wins.