t8rohman / ab_testing_manual

All stuffs we have to know on how to do A/B testing and some alternatives of it.

Thorough Guide to Conducting A/B Tests

A/B tests are widely regarded in the industry as the gold standard for determining the impact of changes to a website or app on user behavior. Their ability to provide reliable results based on robust statistical calculations has made them a favorite among analysts. However, conducting A/B tests involves following a set of procedures to ensure the analysis is sound.

This guide provides comprehensive instructions on conducting A/B tests, covering the following topics:

1. Fundamentals of A/B Testing (basics_abtest)

   • Determining sample size using power analysis
   • Statistical methodologies for calculations, including t-tests and chi-square tests
   • Conducting A/B tests without sample size limitations using Bayesian estimation

2. A/B Testing with Interrupted Time Series / ITS (interruptedtimeseries_its)

   • Conducting A/B tests when dividing traffic into A/B groups is not feasible
   • Incorporating time series components, SARIMAX, into the ITS model
   • Bayesian approach using Bayesian Structural Time Series (BSTS) + ITS