Question 30·Hard·Evaluate Statistical Claims: Observational Studies and Experiments
A beverage company introduced a new energy drink and, one month later, emailed every customer who had purchased the drink from the company’s website. The email asked, “Did the drink improve your focus during the workday?” Out of the 2,300 customers who completed the survey, 82% replied “Yes.” The company then advertised, “More than 80% of people say our drink improves focus.” Which statement best explains why this advertisement may be misleading?
For statistics and survey questions on the SAT, first identify the population the conclusion is about (for example, “all people,” “all students at a school”) and then identify the sample actually used in the study. Ask whether the sample is reasonably representative of that population or whether there is clear bias (such as only volunteers, only customers, or only people from one location). Eliminate choices that talk about irrelevant details (like unused demographic data or adequate sample sizes) and focus on the option that correctly points out the mismatch between the sample and the broad claim being made.
Hints
Focus on who the claim is about
The ad says, “More than 80% of people…” Compare that to who was actually surveyed in the description. Are those the same group?
Think about representativeness, not just size
Even if a survey has many responses, it can still be misleading. Ask yourself: Does the sample accurately represent the larger group mentioned in the claim?
Look for bias in who responded
Consider how the people in the survey were chosen and who chose to answer. Which answer option talks about the way the sample was selected and whether that might cause bias?
Step-by-step Explanation
Identify what the advertisement is claiming
The ad says, “More than 80% of people say our drink improves focus.” This claim is about people in general, not just customers, not just website users, and not just survey responders.
Identify who was actually surveyed
The company emailed only customers who had purchased the drink from the company’s website. Then, from those customers, only the 2,300 who chose to complete the survey were counted in the 82%.
So the 82% comes from a very specific group: people who (1) bought the drink online and (2) decided to respond.
Compare the sample to the population in the claim
The population the ad talks about is all people, but the sample comes from a much narrower and self-selected group.
Customers who bought the drink might already like energy drinks or the brand. Also, people who had strong opinions (especially positive ones) might be more likely to respond to the survey. This means the sample may be biased and not representative of all people.
Check each answer choice against this issue
- Choice A talks about sample size. But 2,300 is a large sample size; that’s not the main problem.
- Choice B talks about the wording (focus vs. energy levels), which doesn’t address the mismatch between the sample and “people.”
- Choice D mentions age groups, but the claim is not about differences between ages.
The only choice that points out that the survey used a non-representative, self-selected sample (only buyers who responded) and therefore can’t justify a claim about all people is: C) Only customers who bought the drink and chose to answer the survey were included, so the sample may not represent all people.