Question 18·Hard·Evaluate Statistical Claims: Observational Studies and Experiments
A city environmental agency mailed a questionnaire to 500 households randomly selected from the city census list to estimate the proportion of households that regularly recycle. Exactly 120 households returned the questionnaire, and 92 of these reported that they recycle regularly.
Which of the following statements must be true?
For questions about surveys and experiments on the SAT, always separate how the sample was chosen from who actually responded or participated. A random selection helps make the initial sample representative, but a low response rate or voluntary participation can introduce bias, especially if certain types of people are more likely to respond. Look out for answers that jump from a sample percentage to a population conclusion or that use absolute language like “must,” “necessarily,” or “unlikely to be biased” without addressing possible nonresponse or self-selection.
Hints
Start with the numbers given
Compute the proportion of the respondents who say they recycle. It is . How does that relate to 76.7% in choice A?
Think about who actually answered
The 500 households were randomly chosen, but only 120 mailed the questionnaire back. Are the 120 respondents guaranteed to be a fair cross-section of all 500, or might certain types of households be more likely to respond?
Pay attention to strong wording
Look for words like “must,” “unlikely to be biased,” and “necessarily biased.” Ask yourself: do we have enough information to be that sure, especially when many people did not respond?
Desmos Guide
Check the sample proportion numerically
Enter 92/120 into Desmos. Note the decimal and convert it to a percentage to see how it compares to 76.7%; this confirms that 76.7% describes only the respondents, not necessarily the whole city.
Step-by-step Explanation
Organize the information and compute the sample proportion
From the 500 households that were mailed the questionnaire, only 120 responded, and 92 of those said they recycle regularly.
The proportion of respondents who recycle is
So 76.7% describes the 120 respondents, not necessarily all households in the city.
Focus on what “must be true” means
The question asks which statement must be true. That means it has to be guaranteed based on the information given, not just plausible or sometimes true.
Be very careful with answer choices that:
- Turn a sample percentage directly into a population percentage, or
- Make strong claims like “unlikely to be biased” or “necessarily biased” without solid justification.
Distinguish random selection from who actually responded
The 500 households were randomly selected from the city census list. That helps make the initial sample representative if they all responded.
But only 120 (24%) actually responded, and those 120 chose to respond. That means:
- The respondents are not guaranteed to be a random subset of the 500.
- Certain types of households (for example, people who care more about the environment) might be more likely to return the questionnaire.
This situation is called nonresponse bias: when a large portion of those selected do not respond, and the responders differ in a systematic way from nonresponders.
Match this reasoning to the correct statement
Now compare the answer choices:
- A) Claims that approximately 76.7% of all households recycle. That takes the respondent percentage and assumes it equals the true citywide percentage. Because of possible nonresponse bias, this is not guaranteed.
- B) Says the estimate is unlikely to be biased just because the 500 were randomly selected. This ignores that only 120 responded; the responding group may be biased.
- D) Says the estimate is necessarily biased downward because nonrecyclers are less likely to respond. We are not told that nonrecyclers are less likely to respond, and even if that were true, it would bias the estimate upward, not downward.
The only statement that must be true, given the low response rate and the possibility that recyclers are more motivated to respond, is:
C) The low response rate may lead to a biased estimate because households that recycle regularly could be more likely to return the questionnaire.