Question 35·Medium·Evaluate Statistical Claims: Observational Studies and Experiments
A company tested two different subject lines for a promotional email. The company randomly selected 2,000 customers; 1,000 received Subject Line A, and 1,000 received Subject Line B. Of those who received Subject Line A, 120 made a purchase, and of those who received Subject Line B, 150 made a purchase.
Based on the results of this experiment, which of the following statements must be true?
I. Among the 2,000 customers in the experiment, the purchase rate was higher for Subject Line B than for Subject Line A.
II. For all of the company’s customers, sending Subject Line B will result in a higher purchase rate than sending Subject Line A.
For questions about conclusions from studies, always separate (1) what actually happened in the sample or experiment from (2) what is being claimed about a larger population or the future. First, do any basic computations (like proportions or averages) to see what the data show for the participants. Then check each statement word-for-word: Is it about just the sample, or about all people? Does the study use random assignment (supports cause-and-effect) and/or random selection (supports generalizing to a population)? Finally, remember that experiments and samples give strong evidence, but very rarely allow you to say something will happen with 100% certainty for every individual; be skeptical of absolute language like "for all" or "will result in" unless it is directly forced by the data and the study design.
Hints
Compare the two subject lines in the experiment
First, find the purchase rate for each subject line using the given numbers: for each group, divide the number of purchases by 1,000.
Focus on what Statement I is actually about
Read Statement I carefully: is it describing the 2,000 customers in the experiment, or all of the company’s customers? Once you know which group it refers to, check whether the given counts support or contradict it.
Think about what we can claim from a sample
For Statement II, ask yourself: does seeing a higher purchase rate for one subject line in a random sample force that subject line to have a higher rate for every customer in the entire population, or does it only give evidence about what is likely?
Use the words "must be true" to limit your choice
"Must be true" means it is guaranteed by the data and setup, not just likely. Eliminate any statement that goes beyond what the experiment actually proves.
Desmos Guide
Compute and compare purchase rates
In Desmos, type 120/1000 on one line and 150/1000 on another line. Compare the two decimal values Desmos shows to see which subject line had the higher purchase rate in the experiment; this will help you judge Statement I.
Recognize the limit of what Desmos can tell you
Notice that Desmos can only calculate these sample rates; it cannot tell you whether the same inequality in purchase rates must hold for all customers in the future. Use logic and your understanding of sampling and experiments to evaluate Statement II.
Step-by-step Explanation
Compute purchase rates for each subject line
For the experiment:
- Subject Line A: 120 out of 1,000 customers made a purchase, so the purchase rate is (12%).
- Subject Line B: 150 out of 1,000 customers made a purchase, so the purchase rate is (15%).
Within this experiment, one subject line clearly had a higher purchase rate than the other.
Interpret Statement I (about the experiment only)
Statement I says: "Among the 2,000 customers in the experiment, the purchase rate was higher for Subject Line B than for Subject Line A."
This statement is talking only about the people who were actually in the experiment (the 2,000 selected customers). You already know exactly how many of them bought something under each subject line, so you can decide whether this statement has to match the data.
Interpret Statement II (generalizing to all customers)
Statement II says: "For all of the company’s customers, sending Subject Line B will result in a higher purchase rate than sending Subject Line A."
This statement is about the entire population of customers, not just the 2,000 in the study. A random experiment with a sample gives evidence about what tends to happen, but it does not guarantee that one subject line will always produce a higher purchase rate for every group of customers or in all future situations.
Think: Does seeing a higher rate in one sample force it to be higher for every possible group and time?
Decide which statements must be true and choose the answer
From the data, among the 2,000 customers in the experiment, Subject Line B had a purchase rate of 15% and Subject Line A had a purchase rate of 12%, so Statement I must be true.
However, the experiment only provides evidence about trends for all customers; it does not prove that Subject Line B will always produce a higher purchase rate for every customer in the full population, so Statement II does not have to be true.
Therefore, the only statement that must be true is I only, which corresponds to answer choice B) I only.