Sampling Bias and Random Assignment
Statistics through experiments · 6 of 16 · CC BY-SA 4.0
A bigger sample can make you more certain about the wrong group.
Let’s return to the town. You could choose residents from the complete list, or save time by measuring people near the basketball court. Would forty people at the court necessarily tell you more about the town than ten chosen from anywhere?
In this fictional town, the court group happens to be taller. Try both selection rules. Increase the sample size and watch whether their estimates tell the same story.
A fictional town of 200 adults
Sample without replacement: nobody is measured twice in a single sample.
Each dot is a resident. Filled dots mark the town-wide sample. Thick outlined dots mark other residents in the basketball-court subgroup.
No sample yet. What do you think the town’s average height will be?
Population mean: 169.4 cm. Measuring all 40 court residents removes uncertainty about that subgroup, but it still does not represent the whole town.
Who had a chance to be included?
The town-wide rule gives every resident an equal chance of selection. The court rule gives most residents no chance at all. That matters because inclusion is related to what we measure: height.
This is selection bias. A selection process can systematically overrepresent some kinds of observations and underrepresent others. More observations from the same restricted group do not fix that mismatch.
Choose forty people. The court sample now includes every member of its subgroup. Its mean is perfectly accurate for that subgroup and still misleading as a description of the whole town.
A moment to think
A different job for randomness
Now suppose forty students try a study app. Students who choose the app score higher afterward. Did the app help?
Perhaps the keenest students chose it. Their existing preparation could explain both their choice and their scores. That alternative explanation is confounding: another factor is tangled up with the comparison we wanted to make.
In the next experiment, the app has exactly zero effect. Students begin with different levels of preparation. Compare letting students choose with assigning them randomly.
Does the study app work?
In this invented class, the app adds exactly zero points to anyone’s score. The already stronger students are more eager to use it. Compare their choice with a random split.
Choose a way to form the groups.
Every random split assigns 20 students to each group. Try several: random assignment balances starting differences on average, not perfectly in every trial.
Random assignment decides which treatment a participant receives. It makes treatment groups comparable on average across possible assignments, so pre-existing differences do not systematically favor a treatment. A particular assignment can still be unbalanced.
Random sampling and random assignment have different jobs. Sampling helps us represent a population; assignment helps us isolate a treatment’s effect among participants. A study can use either, both, or neither.
Random assignment supports causal inference when the study is carried out appropriately—for example, outcomes are measured comparably and participants do not change one another’s treatments. Observational studies can also support causal claims, but need additional design and assumptions to address alternative explanations.
Two design principles
Keep both arrows in view
Population → selection rule → sample
Ask who could be included. A larger sample does not automatically repair a biased selection rule.
Participants → assignment rule → treatment groups
Ask what makes the groups comparable. Random assignment addresses systematic differences before treatment; it does not turn volunteers into a random sample of everyone.
Return to the experiment ↑A moment to think
Words you met
| Term | Meaning |
|---|---|
| Selection bias | Systematic distortion caused by who gets included. |
| Random sampling | Selecting observations by a chance-based rule from a population. |
| Random assignment | Using chance to allocate treatments among participants. |
| Confounding | An alternative factor entangled with the relationship being studied. |
| Association | Values of two variables vary together; that alone does not establish causation. |
Neighbors
- Explore other sampling and experimental designs in Introduction to Data.
Written by June Kim. The conversational approach was inspired by Danielle Navarro’s Learning Statistics with R (CC BY-SA 4.0). The prose, experiments, and questions here were created for this book. This chapter’s text and illustrations are also shared under CC BY-SA 4.0.