Question 42·Hard·Evaluate Statistical Claims: Observational Studies and Experiments
To evaluate a new online algebra game, researchers recruited 120 ninth-grade volunteers from a single high school. The volunteers were randomly assigned to one of two groups:
- Game group (60 students): played the algebra game for 20 minutes each school day for 6 weeks.
- Control group (60 students): continued their usual math homework for 6 weeks.
At the end of the 6 weeks, the average end-of-unit test score of the game group was 8 points higher than that of the control group, and the difference was statistically significant.
Which conclusion is best supported by the design and results of this study?
For SAT questions about study conclusions, first identify whether the study is an observational study or a randomized experiment: if participants are randomly assigned to treatments, you can usually infer causation for those participants. Then check whether the participants were a random sample from a larger population; if not, you should not generalize to all students in a school, district, or country. Finally, scan the answer choices and eliminate any that (1) misuse causation/association based on the study type or (2) make claims about a broader group than the study actually represents.
Hints
Decide what kind of study this is
Ask yourself: Did the researchers just observe students, or did they assign some students to use the game and others not to?
Think about causation vs. association
Once you know whether it is an experiment or an observational study, decide: Are you allowed to draw a cause-and-effect conclusion, or can you only say that two things are associated?
Consider who the results apply to
Were the 120 students a random sample of all students in the school or district, or just volunteers from one school? How does that affect how broadly you can state the conclusion?
Match the answer to the actual group studied
Compare the group described in each answer choice to the group actually in the study, and eliminate any choice that talks about a larger population than the study reasonably supports.
Step-by-step Explanation
Identify the type of study
The researchers recruited volunteers and then randomly assigned them to either play the game or continue usual homework.
Because the researchers actively assigned different treatments to the students, this is a randomized experiment, not an observational study.
Recall what randomized experiments allow you to conclude
Random assignment to groups (like game vs control) is what allows us to make cause-and-effect conclusions about the people in the experiment.
However, these 120 students were just volunteers from a single high school, not a random sample of all ninth-graders at the school or in the district. That means we cannot safely generalize to all students beyond the volunteers.
Use the study type to eliminate answers about association only
Answer choice B says the game is merely associated with higher scores and that no cause-and-effect conclusion can be drawn.
But because this is a randomized experiment with a statistically significant difference between the game and control groups, we can reasonably infer that the game caused the higher scores for the participants. So any answer that refuses all causation is too weak.
Check how far each remaining answer tries to generalize
Answers A and C both talk about all students in bigger groups:
- A: all ninth-grade students at the school
- C: all high-school students in the district
Because the study used volunteers from one school and did not randomly sample from all ninth-graders or from the whole district, we cannot claim the result for all those larger groups. These answers overgeneralize and must be rejected.
The only choice that (1) allows a cause-and-effect conclusion and (2) limits the claim to the actual participants in the study is:
Using the online algebra game caused higher test scores for the ninth-grade students who participated in the study.