Errors, Power, and Multiple Testing

Statistics through experiments · 13 of 16 · CC BY-SA 4.0

Imagine testing the same study procedure in many possible worlds. In one, the new lesson has no effect. In another, it adds five points on average. We get a fresh sample each time and apply exactly the same decision rule.

In an actual study we do not know which world we inhabit. In a simulation we can choose. That lets us count mistakes and learn what a significance threshold does.

Two kinds of mistake

Start with a true difference of zero. Whenever the procedure rejects the no-difference null, it makes a Type I error, or false positive. The preselected threshold α controls that rejection probability under the null when the procedure’s assumptions hold. It is a property of repeated use, not the probability that this particular conclusion is false.

Now choose a real difference of five points. A test that fails to reject makes a Type II error for that alternative. Its probability is called β (“beta”). Power, 1 − β, is the probability of rejecting the null when that specified effect is real.

This activity assumes independent normal measurements with a known population standard deviation of ten points in each group. The two-sided z test therefore has an exact normal reference. Real studies commonly estimate variability and use t procedures; the same ideas about errors and power apply, but their calculations differ.

Make a small signal easier to hear

Hold the true effect fixed and increase the sample size. The estimates become less scattered, so more clear the rejection threshold. Larger effects are also easier to detect. Greater measurement variability makes detection harder.

Lowering α asks for stronger evidence before rejecting. With everything else fixed, this reduces false positives but also reduces power. There is no universal threshold that removes both kinds of mistake.

Plan around the smallest effect that would change a decision, not the largest effect you hope to see. Estimate plausible variability, allow for missing outcomes, and choose a sample size before collecting results. A study can have enough power to detect a trivial effect, yet answer the practical question poorly.

What if we keep looking?

Open the multiple-tests panel. With twenty independent tests of true null hypotheses, each at α = 0.05, the probability of at least one false positive is about 64%. None of the individual tests changed; we gave chance more opportunities to produce a headline.

A simple Bonferroni correction tests each of m prespecified hypotheses at α/m. It controls the probability of any false rejection at no more than α even when the tests are dependent, provided the individual p-values are valid. It can be conservative. Other methods answer related questions, such as controlling the expected fraction of false discoveries.

Repeatedly checking one accumulating dataset and stopping at the first small p-value creates a related problem. A fixed-sample test assumes its planned stopping rule. Sequential designs can allow interim looks, but require methods designed for them. Exploratory analysis remains useful; label it and seek new data for confirmation.

After a study, the effect estimate and confidence interval usually communicate uncertainty more directly than plugging the observed effect into a new power calculation. Power earns its place most clearly when deciding what study to run.

Words you met

Written by June Kim. The conversational approach was inspired by Danielle Navarro’s Learning Statistics with R (CC BY-SA 4.0). The prose, experiments, and questions here were created for this book. This chapter’s text and illustrations are also shared under CC BY-SA 4.0.