Question 31·Easy·Evaluate Statistical Claims: Observational Studies and Experiments
To estimate the proportion of adults in a town who support building new bike lanes, a researcher mailed questionnaires to 1,000 households selected at random from the town directory. Only 120 households returned the questionnaire, and 90 of those respondents said they were in favor of the project.
Which statement about the researcher’s conclusion is most accurate?
For survey and experiment questions on the SAT, focus first on how the sample was actually obtained and who ended up in the sample, not just on the numbers. Distinguish between those who were selected and those who responded, and watch for nonresponse or voluntary response bias. Evaluate answer choices by asking whether they correctly address representativeness and possible bias, and be skeptical of any claim that a certain sample size alone “guarantees” accuracy or that a sample is invalid just because it isn’t enormous.
Hints
Clarify what the 75% refers to
First figure out what group the 75% comes from: is it all adults in the town, all 1,000 households contacted, or just the households that returned the questionnaire?
Consider who actually makes up the sample
Does every household that was mailed a questionnaire end up in the sample, or only the households that chose to respond? How might that difference matter?
Think about representativeness, not just size
Even if a number seems large, ask: are the people in the sample likely to reflect the attitudes of the whole population, or could certain types of people be more likely to respond?
Check each choice for common survey reasoning mistakes
Look for choices that confuse "who was contacted" with "who actually responded," or that make extreme claims about sample size or guarantee perfect accuracy.
Desmos Guide
Compute the sample proportion that supports the project
In the Desmos calculator, type 90/120 and press Enter. Note the decimal or percentage shown—this is the proportion of respondents (not all contacted households) who said they were in favor. Use this to remind yourself the conclusion is based only on those who responded, and then reason about whether that group necessarily represents all adults in the town.
Step-by-step Explanation
Identify what the researcher actually measured
From the 120 households that returned the questionnaire, 90 were in favor of the project.
So the sample proportion from the respondents is
This tells us that 75% of the people who responded support the project. The question, however, is whether the researcher can trust this as an estimate for all town adults.
Distinguish between who was contacted and who actually responded
The 1,000 households were chosen at random, which is good. But only 120 of them responded.
Those 120 are volunteers who chose to mail the questionnaire back. People who care strongly (for or against) the project might be more likely to respond than people who are indifferent.
So the real sample is not all 1,000 households, but just the 120 that responded—and that group might not represent the whole town.
Think about possible bias (nonresponse / voluntary response)
When many people do not respond, the sample can suffer from nonresponse bias or voluntary response bias.
In that case, even if the original 1,000 were selected randomly, the final group of 120 respondents can be systematically different from all adults in the town.
That means the conclusion based on those 120 responses might not accurately reflect the true proportion of adults in the town who support the project.
Match the reasoning to the answer choices
Now compare this idea to the choices:
- (A) incorrectly assumes that random selection of the original 1,000 guarantees that the respondents are representative.
- (C) is too extreme: 120 is not “too small for calculating any percentage.”
- (D) focuses on the 1,000 contacted, but the actual sample is only the 120 who responded; contacting 1,000 does not “guarantee” accuracy.
The only choice that correctly points out the real issue—that the 120 respondents may not represent all households in the town—is:
B) The conclusion may be unreliable because the 120 households that responded might not represent all households in the town.