Thinking critically about numbers
Every day we see statistical claims: "4 out of 5 dentists recommend…", "Crime up 50%!", "Studies show…". Some are solid, and some are misleading statistics. Knowing what questions to ask helps you judge them.
Questions to ask
| Question | Why |
|---|---|
| Who collected the data, and who paid? | possible conflicts of interest |
| How was the sample chosen? | random vs convenience or voluntary |
| How big was the sample size? | small samples are unreliable |
| What exactly was measured or asked? | wording and definitions matter |
| What’s the comparison? | percentages need a base |
| Is it correlation or causation? | association doesn’t prove cause |
| Is the graph honest? | scales and axes can mislead |
| What’s missing? | cherry-picking hides inconvenient data |
Relative vs absolute risk
"A new medicine cuts the risk of a disease by 50%!"
- Without it: 2 in 1,000 get the disease (absolute risk 0.2%).
- With it: 1 in 1,000 (0.1%).
The relative risk dropped 50%, but the absolute risk only dropped 0.1 percentage points. Both are true; the absolute number gives the fuller picture.
Percentages without context
"Sales up 200%!" From 1 unit to 3? Or from 10,000 to 30,000? Always ask for the base.
"Crime doubled" in a small town might mean from 2 incidents to 4.
Averages that mislead
"The average salary at our company is $95,000." If the CEO earns $2 million and most workers earn $45,000, the mean is pulled up; the median tells a different story.
Misleading graphs
- y-axis not starting at 0 exaggerates differences.
- Uneven intervals distort trends.
- 3D pie charts warp slice sizes.
- Cherry-picked time ranges hide the full trend.
Survey tricks
- Loaded questions: "Do you support protecting children by…?"
- Voluntary response: online polls attract strong opinions.
- Tiny samples: "4 out of 5 dentists" might mean exactly 5 dentists were asked.
Margin of error
A poll showing Candidate A at 48% and B at 46% with a ±3% margin is a statistical tie, not a lead.
Worked examples
Example 1: "People who drink green tea live longer." What’s missing? Possible lurking variables (overall diet, exercise, income); likely observational.
Example 2: A claim based on 12 volunteers. Concern? Small, non-random sample.
Example 3: Risk falls from 4% to 3%. Relative and absolute changes? Relative: 25% decrease; absolute: 1 percentage point.
Common mistakes
- Accepting claims because they include numbers.
- Confusing relative and absolute risk.
- Assuming the source is neutral.
Try it yourself
- A bar graph’s y-axis starts at 90. What could be misleading?
- "Our toothpaste reduces cavities by 30%." What would you ask?
- Two candidates: 51% vs 49%, margin ±4%. Is one clearly ahead?
- Risk rises from 1 in 10,000 to 2 in 10,000. Relative increase? Absolute increase?
Answers: 1) Small differences look huge. 2) e.g. compared with what, in how many people, who funded the study? 3) No; it’s within the margin of error. 4) 100%; 0.01 percentage points
Quick recap
- Question every statistical claim: source, sample, wording, comparison.
- Distinguish relative risk and absolute risk.
- Watch for cherry-picking, misleading graphs and small sample sizes.