Introduction
A/B testing is about making decisions with a controlled level of risk. Kameleoon supports three statistical approaches for interpreting your results: Frequentist, Bayesian, and sequential testing. This guide compares the three approaches so you can choose the right one for your team’s experimentation goals and risk tolerance.Available approaches
Each approach carries its own advantages and disadvantages and uses a distinct methodology.Fixed sample Frequentist
The Fixed sample Frequentist method requires you to define your hypothesis and sample size before you launch a test. If you follow that plan without changing it based on interim results, you get optimal control over statistical risk.Pros
- Delivers maximal statistical power.
- Serves as the most widely used method for randomized controlled trials.
Cons
- Requires a rigid, predefined sample size and test duration.
Best for
- Mature experimentation teams that understand the risk of deviating from the method.
Bayesian
The Bayesian method offers more flexibility by incorporating existing knowledge into the analysis through prior probabilities. This approach works best when ample prior data is available to calculate a posterior probability.Pros
- Lets you leverage your existing knowledge.
- Gives you access to key data based on the estimated distribution.
Cons
- Can lead you in the wrong direction if the prior data is incorrect.
Best for
- Experimentation teams that are experienced with Bayesian methods and probability distributions.
Sequential
Sequential testing solves the problem of premature experiment termination caused by interim results that deviate from expectations. Unlike fixed-sample-size tests, sequential testing lets you make decisions dynamically as data accumulates. This flexibility can help you reach conclusions sooner, but it can also produce a less precise effect size estimate.Pros
- Adapts to accumulating data without requiring you to estimate a sample size in advance.
- Provides a valid confidence interval at any point during the experiment’s runtime.
Cons
- Delivers lower statistical power than fixed sample methods.
Best for
- Fast-moving teams that want more flexibility than the fixed sample framework allows and are willing to trade some statistical power for it.
To get the benefits of both approaches, combine sequential and fixed sample size methodologies. Use sequential testing to stop early when results become significant, and use fixed sample sizes to keep your findings statistically rigorous. Setting thresholds for early stopping and validating findings with the fixed sample method lets you accelerate insights without compromising reliability.
Complementary techniques
Multiple testing correction
Testing multiple variations simultaneously accelerates the overall pace of experimentation. However, to maintain statistical integrity when you do this, apply correction methods that reduce the risk of false positive results.CUPED
CUPED reduces the sample size an experiment needs by leveraging pre-experiment data, which improves statistical power and accelerates experimentation. Combine CUPED with a fixed sample size methodology to detect significant results early while maintaining statistical power. CUPED works best when:- Your experiment includes returning visitors.
- You’ve run many experiments in Kameleoon.
- Goal conversions before the experiment starts correlate with conversions during the live experiment.