Question 37·Easy·Evaluate Statistical Claims: Observational Studies and Experiments
A nutrition company wanted to test whether its new energy drink improves endurance. The company posted an online advertisement asking for volunteers who were interested in boosting their athletic performance. Sixty volunteers responded, drank the energy drink every day for two weeks, and then reported their endurance levels. Forty-five of the volunteers said their endurance had improved.
Which statement best evaluates the validity of the company’s conclusion that the drink increases endurance for the general population?
For questions about evaluating a study or survey, first summarize in your own words how participants were chosen and what was measured. Then, look for common design issues: biased or self-selected samples (not representative), lack of random assignment, missing control/placebo groups, and overly strong conclusions (like "proves" or "for all people"). Eliminate answer choices that talk about problems not supported by the description (such as claiming the sample is too large) and choose the option that best matches the actual weakness in how the data were collected or how the conclusion is stated.
Hints
Look closely at how participants were chosen
Pay attention to the phrase that describes how the 60 people ended up in the study. Were they picked randomly, or did they choose themselves to participate?
Think about what you need to say something about "the general population"
Ask yourself: if you want to make a claim about most people, what has to be true about the group you actually tested? Consider whether this group fits that requirement.
Eliminate answers that talk about the wrong problem
Decide whether the main concern here is the number of participants, the type of people included, or whether the result is stated too strongly. Cross out choices that focus on something that is not really an issue in this study.
Step-by-step Explanation
Understand what the study did and concluded
The company:
- Posted an online ad asking for volunteers "interested in boosting their athletic performance."
- Sixty volunteers responded and drank the energy drink daily for two weeks.
- Afterward, they self-reported whether their endurance improved.
- Forty-five said their endurance improved.
From this, the company concluded that the drink increases endurance for the general population.
Identify potential design problems
Look for issues that could make the conclusion weak or invalid:
- Self-selection: People chose to volunteer, and specifically those interested in improving performance.
- No random sampling: Participants were not randomly chosen from all people.
- No control group or placebo: There is nothing to compare against people who did not drink it.
- Self-reported results: People just said whether they felt better; this may be biased.
Any of these could limit how far we can trust or generalize the result.
Focus on what "for the general population" requires
To claim something about "the general population," the sample should:
- Be representative of that larger population (for example, chosen in a way that doesn’t favor a particular type of person).
- Not be limited to a special group, such as only highly motivated athletes responding to an ad about boosting performance.
If the sample systematically differs from typical people, then results from that sample may not apply broadly.
Match each answer choice to the real issue
Now compare each choice to the issues above:
- Choice A says the study proves the drink works for all people—this is far too strong and ignores the design problems.
- Choice C suggests including only people who already use other energy drinks, which would actually make the sample less like the general population.
- Choice D says the sample size is too large, but a larger sample is generally better, not worse.
Only Choice B correctly points out the key problem: because the participants were volunteers responding to a specific ad, they may not represent the general population, so the study’s results cannot be safely generalized to everyone.