On the pitfalls of A/B testing
You’ve (hopefully) heard many things about A/B testing. People rave about it, and with good reason: It provides you with a solid framework on which to evaluate how a change you’ve made has affected your metrics. Gut feeling isn’t always enough, and hard numbers help tremendously in decisions. However, most people (and software packages) make a grave mistake when A/B testing. This post will help you recognize and avoid it.
In case you’re unfamiliar with A/B testing, here’s a small run-down (it might get a bit technical, but shouldn’t be too bad):
