When data goes wrong
Bias is a systematic error: results lean consistently in one direction. Unlike random error, bias doesn’t shrink with a bigger sample.
Types of bias
| Type | What happens | Example |
|---|---|---|
| Sampling bias | the sample isn’t representative | surveying only gym members about exercise habits |
| Non-response bias | non-responders differ from responders | busy people ignore a long survey about free time |
| Response bias | people answer inaccurately | under-reporting screen time or overstating exercise |
| Measurement bias | tools or methods are faulty | a scale that reads 2 kg too heavy |
| Voluntary response bias | only motivated people reply | online reviews skew to very happy or very angry customers |
Question wording
A leading question pushes respondents:
| Biased | Neutral |
|---|---|
| "Don’t you agree that homework is a waste of time?" | "How useful do you find homework?" |
| "Should the city waste money on a new arena?" | "Should the city build a new arena?" |
Other wording problems:
- Loaded words: "reckless", "heroic"
- Double-barrelled questions: "Do you like the food and the service?" (two questions in one)
- Confusing negatives: "Do you not disagree…?"
- Missing options: no "neither" or "unsure" choice
Response bias causes
- Social desirability: people give answers that make them look good.
- Interviewer effects: tone or presence influences answers.
- Sensitive topics: people hide the truth about income, illegal activity or health.
- Order effects: earlier questions affect later answers.
Anonymous surveys can reduce some response bias.
Reducing bias
| Strategy | Helps with |
|---|---|
| random sampling | sampling bias |
| follow-up reminders, short surveys | non-response bias |
| neutral wording, pilot testing | leading questions |
| anonymity | response bias |
| calibrated instruments | measurement bias |
Case study
A magazine asks readers to mail in opinions on a political issue and gets 20,000 replies. Problems: voluntary response, readers aren’t representative of the public, and strong opinions dominate. A random sample of 1,000 people would be more trustworthy.
Worked examples
Example 1: A teacher asks students aloud, "Who cheated on the test?" Bias? Response bias (social desirability).
Example 2: "Do you support lower taxes and better schools?" Problem? Double-barrelled question.
Example 3: Surveying park visitors about funding parks. Bias? Sampling bias (park users favour parks).
Common mistakes
- Thinking a large sample removes bias.
- Writing questions with built-in opinions.
- Ignoring who didn’t respond.
Try it yourself
- Rewrite neutrally: "Wouldn’t it be great to have a longer lunch?"
- A phone survey during work hours reaches mostly retirees. Type of bias?
- Only 15% of mailed surveys come back. What bias might occur?
- A thermometer reads 1 °C too high. Type of bias?
Answers: 1) e.g. "Would you prefer a longer, shorter or the same lunch period?" 2) sampling bias 3) non-response bias 4) measurement bias
Quick recap
- Bias systematically distorts results.
- Watch for sampling bias, non-response bias, response bias and measurement bias.
- Avoid leading questions; use random samples and neutral wording.