Question 50·Medium·Evaluate Statistical Claims: Observational Studies and Experiments
A national news website included an online poll asking visitors whether they support a proposed change to voting laws. Of the 500 visitors who responded, 68% said they support the change. On the basis of this result, the website reported, "About two-thirds of adults in the country support the new voting law."
Which of the following best explains why the website's conclusion may not be valid?
For questions about the validity of survey or experiment conclusions, first identify the population being claimed about and the sample actually observed. Ask: Was the sample randomly selected and is it likely representative of the population, or is it a convenience/voluntary sample (like an online poll or people who choose to respond)? Then quickly test each answer choice: eliminate options that focus on tiny numerical differences or claim the sample size is too small when it is actually reasonable, and prioritize explanations that point to bias in how participants were selected or lack of randomization, since those are the most common reasons conclusions are not valid.
Hints
Think about who was surveyed
Focus on who is answering the poll. Are these people chosen randomly from all adults, or are they a particular group?
Consider what makes a sample trustworthy
Ask yourself: what conditions must a sample meet so that we can confidently say its results apply to the whole population?
Compare each choice to the situation
For each answer choice, ask: does this describe a realistic problem with how the poll was conducted (who was in it and how they were chosen), or is it about something else like exact percentages or sample size?
Desmos Guide
Compare 68% to two-thirds numerically
Type 0.68 and 2/3 into Desmos (on separate lines). Look at their decimal values and note how close they are. This helps you judge whether saying “about two-thirds” is a reasonable description of 68%.
Estimate a typical margin of error for 500 people (optional)
If you want to explore margin of error, you can enter sqrt(0.68*(1-0.68)/500) in Desmos to see the approximate standard error of the sample proportion. Then multiply that result by 2 or so to get a rough sense of a typical margin of error, and compare it to the difference between 0.68 and 2/3. This shows that the small numerical difference is not the main concern; think instead about how the sample was chosen.
Step-by-step Explanation
Understand what the website is claiming
The poll result is based on 500 website visitors, and 68% of them said they support the proposed voting law. The website then stated that about two-thirds of adults in the country support the law. This is a generalization from a sample to an entire population.
Identify the type of sample used
The people in the poll are visitors to a national news website who chose to respond to an online poll. This is a voluntary online sample, not a randomly selected sample of all adults in the country. People who visit that website and decide to answer the poll may differ from adults in general (for example, in age, political interest, or views on voting laws).
Check what makes a generalization valid in statistics
To make a reliable claim about an entire population, you generally need a sample that is:
- Randomly selected from the population, and
- Representative of that population (similar in key ways, not systematically different).
Even a fairly large sample can give misleading results if it is biased (for example, only people with strong opinions choose to respond, or only people who visit a certain website are included).
Evaluate why the specific answer choices are weaker
Now briefly consider the other answer choices:
- A margin of error is never exactly 0% for a sample, but the website never claimed it was, so this doesn’t directly explain the problem.
- is very close to (about ), so saying “about two-thirds” is fine and not the main issue.
- A sample of 500 can actually be large enough for estimating opinions if it’s representative.
The core problem is that the website is treating a non-representative, voluntary sample of website visitors as if it represented all adults in the country. Therefore, the best explanation is: “The sample of website visitors may not be representative of all adults in the country.”