Comparing Proportions

Statistics through experiments · 11 of 16 · CC BY-SA 4.0

A website offers two versions of a sign-up page. Each visitor sees either A or B, assigned at random, and either signs up or does not. We want to know whether B changes the probability of signing up.

In this simulator you can see what a real experimenter cannot: the probabilities that generate the outcomes. Version A has a 30% sign-up probability. Choose B’s advantage, draw a study, and see how closely the observed difference follows it.

Keep three quantities separate

The population proportion is the underlying probability of success for a group. The sample proportion is successes divided by observed trials. A hat marks the estimate: p̂ is read “p-hat.” The effect estimate here is p̂_B − p̂_A.

If A gets 30% and B gets 40%, the difference is ten percentage points. The relative increase is ten divided by thirty, about 33%. Both describe the same change, but they answer different questions. A report should say which it uses.

For the activity’s model, visitors are independent, each appears once, and the success probability stays constant within each group. In practice, repeat visits, shared households, missing outcomes, or one visitor seeing both versions can require a different design or analysis.

Build a world where the labels do not matter

Under a sharp null hypothesis of no treatment effect for any visitor, each recorded response would be the same under A or B. We can keep the responses and repeatedly reassign the group labels, preserving the original group sizes. Every shuffle gives a difference that random assignment alone could produce.

That collection is a randomization distribution. It centers near zero. We compare our observed difference with it, counting shuffled differences at least as far from zero in either direction. This is a two-sided test: the alternative allows B to help or hurt.

Choose the direction before seeing the data. Choosing a favorable one-sided test afterward changes the error rate. Our simulated p-value includes a small correction: add one to both the extreme count and the number of shuffles. A finite simulation then cannot misleadingly report zero.

How much might the rate change?

The p-value asks about compatibility with the null. A confidence interval asks about the precision of the estimated difference. The activity shows a large-sample interval only when each group has at least ten successes and ten failures. That is a screening rule for this approximation, not a guarantee that any study passing it is sound.

Its standard error uses the two observed proportions separately. A normal-based test of equal proportions often instead pools them under the null. Our shuffle test and approximate interval also use different procedures, so their conclusions need not match exactly at a cutoff. For sparse counts, use an appropriate exact or simulation-based method rather than forcing this approximation.

Try setting the true advantage to zero and generating several studies. Some look more impressive than others. That variation is why a single low p-value is evidence within a model, not a certificate of truth.

Words you met

Written by June Kim. The conversational approach was inspired by Danielle Navarro’s Learning Statistics with R (CC BY-SA 4.0). The prose, experiments, and questions here were created for this book. This chapter’s text and illustrations are also shared under CC BY-SA 4.0.