Question 36·Hard·Evaluate Statistical Claims: Observational Studies and Experiments
A school district wanted to determine whether its new voluntary after-school tutoring program improves students’ mathematics performance. After one semester, the district compared the mean standardized math test score of students who chose to attend the tutoring sessions with the mean score of students who did not attend. The students who attended scored, on average, 15 points higher, and the district concluded that the tutoring program caused the improvement.
Which of the following best explains why this conclusion may not be warranted?
For questions about statistical conclusions, first identify the study type: Was there random assignment (experiment) or just observation of existing groups (observational study)? If participants self-select into groups or are not randomly assigned, be suspicious of any strong cause-and-effect claim, because groups may differ in important ways beforehand (confounding variables). On the SAT, scan the answer choices for the one that points out this design flaw or a clear alternative explanation for the results, rather than choices that add new unsupported details or focus on irrelevant measurements.
Hints
Consider whether this was an experiment or an observational study
Ask yourself: Were students randomly assigned to attend or not attend the tutoring program, or did they choose for themselves? How does that affect what we can conclude?
Think about other possible explanations for higher scores
Imagine why students who attend extra tutoring might already differ from those who do not, even before the program starts. How could those differences affect the averages?
Match the issue to an answer choice
Look for the answer that points out a problem with comparing these two groups or with claiming that the program caused the difference, rather than bringing up unrelated data issues or different subjects.
Step-by-step Explanation
Identify what kind of study this is
Notice that the tutoring program is voluntary: students chose whether or not to attend. There was no mention of students being randomly assigned to tutoring or no tutoring.
That means this is an observational study, not a randomized experiment.
Recall what is needed to justify a cause-and-effect conclusion
To say that one thing caused another (here, that the tutoring caused higher math scores), we usually need a well-designed experiment.
In a good experiment about a program like this, students would be randomly assigned to either attend tutoring or not, so that, on average, the two groups are similar in all ways except for the treatment (the tutoring). That way, differences in outcomes are more likely to be due to the treatment itself.
Think about how self-selection can bias the comparison
Because students self-selected into the tutoring group, it is very possible that students who chose to attend were already different from those who did not attend.
Ask yourself: What kinds of differences might exist between students who sign up for extra help and those who don’t? How could those differences affect math scores even if the program had no effect at all?
Connect the key design flaw to the answer choice
A major flaw is that students who chose to attend tutoring may already have been more motivated, more responsible, or higher-achieving in math before the semester began. These pre-existing differences could explain why their average scores are 15 points higher, even without any effect from the tutoring program.
The best answer is the one that points out this possibility of pre-existing differences between the groups (a confounding factor), which means the score differences might not be attributable to the program itself. That is exactly what choice D states: “Students who chose to attend the tutoring sessions may have been more motivated or higher-achieving before the semester began, so differences in scores might not be attributable to the program.”