Question 4·Easy·Evaluate Statistical Claims: Observational Studies and Experiments
A local newspaper wanted to estimate the proportion of adult residents in the city who support the construction of a new public library. To collect data, the newspaper posted an online poll on its website and invited readers to respond. A total of 2,300 people completed the poll, and 68% of them indicated support for the new library.
Which of the following best describes a concern about using these poll results to draw a conclusion about all adult residents of the city?
For survey and experiment questions on the SAT, quickly identify (1) the population the researchers care about and (2) who was actually in the sample and how they were chosen. Look for red flags like voluntary response (people choose to participate), convenience samples (easy-to-reach people), or restricted groups that leave out parts of the population. Distinguish between issues of sample size and sampling bias: a large but biased sample still gives unreliable conclusions, while a smaller but well-randomized sample can be strong. Then pick the option that best describes the main threat to representativeness and validity.
Hints
Clarify the goal of the poll
Ask yourself: Who is the poll trying to describe? Then, who actually answered the poll?
Look at how the sample was collected
Pay close attention to the words describing how the poll was conducted (for example, “posted an online poll” and “invited readers to respond”). What does that tell you about who ended up in the sample?
Decide what the main problem is
Is the biggest issue that there are too few people, that there is no error at all, that the poll should be more narrowly focused, or that the people who responded might not be typical of the whole population?
Step-by-step Explanation
Identify the population and the sample
First, separate who we care about from who was actually surveyed.
- Population of interest: all adult residents in the city.
- Actual sample: 2,300 people who visited the newspaper’s website and chose to respond to the online poll.
Any concern we choose must focus on whether this sample gives trustworthy information about all adult residents.
Focus on how the sample was obtained, not just its size
Notice how the sample was collected:
- The poll was posted online on the newspaper’s website.
- People were invited to respond; anyone visiting could choose to participate or ignore it.
This is called a voluntary (self-selected) response sample. People with strong opinions (for or against the library) are more likely to answer, and people who don’t read the website or don’t go online at all are automatically left out. This raises a major concern about bias and representativeness, not about the sample being too small.
Match the correct concern to the answer choices
Now compare each choice to the situation:
- One choice points out that people who chose to respond to an online poll might not represent all adult residents, creating potential bias.
- Another claims the sample is too small, but 2,300 is actually quite large.
- Another says the margin of error is exactly 0%, which is never true in a sample survey.
- Another suggests surveying only people who live near the library site, which would make the sample more biased.
The best description of the real concern is: A) The results may be biased because the respondents were self-selected and may not represent all adult residents.