conversionista / can-i-test-that

Conversionista's Experiment Feasibility Calculator a.k.a Can I test that?

So, how long should you run your A/B test?

There are 4 pieces of information that we need to know in order to answer that quesion:

• Conversion Rate of original page
• Number of variations to test (more variations you test, more traffic you need)
• Average daily traffic on your site (not really needed, optional)
• What % difference in conversion rate do you want to detect (if you want to detect even the slightest improvement, it will take much longer)

The last point is really, really hard to esitmate. So we buildt this tool.

The graph above is taken from an excellent book called Statistical Rules of Thumb. Luckily, the chapter on estimating sample size is available to download freely. Another excellent source to get more information on sample size estimation for A/B testing is Microsoft’s paper: Controlled Experiments on the Web: Survey and Practical Guide.