lacunary - Mathnotes

A/B Testing

Definition: A/B Test \@{ab-test}

An A/B test is a two-sample statistical inference problem. When we're running an A/B test, we have two groups of users, group A and group B.

We change some functionality for group A and leave it stable for group B. We want to see if there is a statistically significant difference between group A and group B for some metric.

TODO: finish this. Formalize the two-sample setup in terms of the two random variables X and Y, and describe the CUPED variance-reduction technique the page is named for.