Question 39·Hard·Evaluate Statistical Claims: Observational Studies and Experiments
In a school district that has exactly 10 elementary schools, a researcher wanted to test the effectiveness of a new reading program for 4th-grade students. The researcher randomly selected 5 of the 10 schools to implement the program for one full year; the remaining 5 schools continued with their usual reading curriculum. At the end of the year, every 4th-grade student in the district took the same standardized reading assessment. The average score of students at schools using the new program was significantly higher than the average score of students at the other schools.
Which conclusion is best supported by the results of this study?
For SAT questions about statistical studies, first decide whether the study is observational or an experiment with random assignment; only randomized experiments justify causal language. Next, carefully identify who was actually studied (grade, school/district, region) and only accept answer choices whose scope matches that group. Eliminate any option that (1) uses causal wording for an observational study or (2) generalizes the results beyond the population actually involved in the study (such as to other districts, the whole state, or the entire country).
Hints
Observational study or experiment?
Look at how the schools ended up using the new reading program. Were they just observed as they were, or were some schools assigned to use the new program and others to keep the usual one?
Association vs. causation
Once you know whether this is an observational study or a randomized experiment, think about what kind of conclusion is allowed: can we only say two things are related, or can we say one caused a change in the other?
Scope of the conclusion
Focus on who was actually studied. Were the students from one school district, from the whole state, or from across the country? Which answer choices go beyond the group that was actually in the study?
Eliminate overgeneralizations
Cross out any choices that make claims about students who were not part of the study (for example, different grades, other districts, the entire state, or the whole country). Then decide, among what remains, whether the study design supports a causal statement.
Step-by-step Explanation
Identify the type of study
Read how the study was done: 5 of the 10 schools were randomly selected to implement the new reading program, and the other 5 continued their usual curriculum. This means the researcher assigned schools to different conditions (new program vs usual program), rather than just observing what schools already did. That makes this a randomized experiment, not just an observational study.
Recall what randomized experiments allow you to conclude
In a randomized experiment, random assignment helps balance out other differences between groups (like previous achievement, teacher quality, etc.), so any consistent difference in the outcome (test scores) can reasonably be attributed to the treatment (the new program). Therefore, randomized experiments can support causal conclusions about the subjects who were part of the experiment.
Determine the population the results apply to
The study involved:
- All 10 elementary schools in one school district.
- All 4th-grade students in that district, since every 4th grader took the same standardized reading assessment.
The data tell us about 4th-grade students in this one school district, not about the entire state or the whole country, and not about other grade levels.
Match the correct scope (who) and strength (causal vs. associated) of the conclusion
Because this is a randomized experiment, a causal conclusion is appropriate.
Because the experiment was done only with 4th-grade students in this school district, the conclusion should be limited to that group and should not be extended to all 4th graders in the state or to students nationwide.
The only choice that both (1) states a causal effect and (2) limits the claim to 4th-grade students in this school district is:
The new reading program caused higher average reading scores for 4th-grade students in this school district.