When NOT to run an experiment

Experiments are powerful — and also slow, expensive (in attention, if not money), and easy to misuse. Before you start one, ask whether you actually need it. Three cases where you don't:

1. The answer is obvious

Don't A/B test fixing a typo. Don't A/B test a button that was unclickable. Don't A/B test the dark-pattern unsubscribe flow you already know is wrong. Experiments are for genuine uncertainty — "will users prefer the simpler form or the richer one?" — not for confirming what you already know.

2. You don't have enough traffic

At 50 signups per arm per day, detecting a 5% lift takes about three weeks. At 5 per arm per day, it takes the better part of a year. If you're early-stage, the right tool is a flag (ship it to everyone, watch the metric, decide in a week) — not an experiment that won't conclude before your priorities change.

3. The metric is wrong

You're A/B testing button copy. The metric is "click-through rate on the button." The winner gets +12% clicks — and -3% retention. Whoops: you optimized for the easy-to-measure thing, not the thing you actually wanted. Pick metrics that survive the seven-day check.

When TO run one

Pricing changes. Onboarding flows. Anything where the cost of being wrong-at-scale is higher than the cost of waiting three weeks. That's when the experiment pays for itself.