A/B tests are widely regarded in the industry as the gold standard for determining the impact of changes to a website or app on user behavior. Their ability to provide reliable results based on robust statistical calculations has made them a favorite among analysts. However, conducting A/B tests involves following a set of procedures to ensure the analysis is sound.
This guide provides comprehensive instructions on conducting A/B tests, covering the following topics:
-
Fundamentals of A/B Testing (basics_abtest)
- Determining sample size using power analysis
- Statistical methodologies for calculations, including t-tests and chi-square tests
- Conducting A/B tests without sample size limitations using Bayesian estimation
-
A/B Testing with Interrupted Time Series / ITS (interruptedtimeseries_its)
- Conducting A/B tests when dividing traffic into A/B groups is not feasible
- Incorporating time series components, SARIMAX, into the ITS model
- Bayesian approach using Bayesian Structural Time Series (BSTS) + ITS