Learning Science · Effects, uncertainty and context
An effect always has an outcome and a comparison
Read an effect as a difference between specified conditions on a specified outcome.
Use the examples to connect this lesson with the earlier topics. No specialist equipment is needed.
Back to the chapterAfter reading, you can
- Recover the missing comparator in an effectiveness claim.
- Distinguish a difference in points from a relative percentage.
- Keep the direction and timing of an outcome visible.
An effect has a reference point
A course advertises a “large learning gain.” Compared with what? The learner’s initial score, a waiting list, ordinary teaching or another carefully designed method could each produce a different number. An effect estimate is meaningful only with its outcome, comparison, population and timing.
A higher test score can indicate a better measured outcome, but a higher error count indicates a worse one. Always identify which direction represents improvement before reading a graph or a summary.
| Expression | Calculation | Meaning |
|---|---|---|
| Test-point difference | 8 − 6 = 2 | Two more correct answers on this ten-item test |
| Percentage-point difference | 80% − 60% = 20 percentage points | Difference as a share of the test maximum |
| Relative increase from B | (8 − 6) / 6 ≈ 33% | Difference relative to B’s mean |
Keep the scale attached
In a constructed example, one group averages 8 correct answers out of 10 and another averages 6. The difference is 2 answers, or 20 percentage points on that scale. Relative to 6, the increase is about 33 percent. These are different descriptions of the same arithmetic, and neither is a universal “33 percent improvement in learning.”
The item set determines what those points represent. Two additional definitions recalled are not interchangeable with two additional complex problems solved. Inspect the tasks before interpreting the size of the difference.
Change the comparator, change the question
A new practice routine may outperform no practice while matching an existing routine. That can still matter if the new routine is cheaper or easier to access. Conversely, beating a weak comparator does not show that the programme is the best available option.
For a practical decision, compare the realistic alternatives. Include the time, support and resources that accompany them. If the study evaluates a whole package, use the package as the subject of the claim.
Locate the estimate in a design
A numerical difference can be calculated for any groups. Whether it estimates a causal effect depends on assignment, implementation, missing data and analysis. A precise-looking number does not repair a confounded comparison.
Write the conclusion in one sentence with the relevant conditions. For example: “Under the stated allocation and testing arrangement, this group recalled more of the trained terms after one week than the equal-time restudy group.” Then ask which extension remains untested.
Worked example
Two headlines for one constructed comparison
How can the same hypothetical vocabulary results be reported as points, percentage points and relative change?
- A hypothetical vocabulary activity yields averages of 8/10 and 6/10 on a later test. “Two more answers correct on this test” preserves the direct scale. “Learning increased by a third” conceals the denominator and broadens the outcome.
- A careful report gives group means, the difference and uncertainty, describes the terms and test, and identifies the comparison activity. It can then discuss whether the difference would matter for the actual learning goal.
Means of 8/10 and 6/10 differ by 2 test points, or 20 percentage points of the test maximum. Relative to 6/10, the increase is about 33%; that denominator does not make this “33% more learning” across other outcomes.
Try it yourself
Think through each question before opening the answer.
Is an increase from 40% to 50% ten percent or ten percentage points?
Show the answer and reasoning
It is ten percentage points; relative to 40%, it is a 25% increase. State which quantity you mean.
Percentage points subtract the two percentages. Relative change divides that difference by the starting percentage: (50−40)/40 = 0.25. The two numbers answer differently scaled questions.
Why is outperforming no extra practice insufficient to isolate a special method?
Show the answer and reasoning
The comparison also changes practice opportunity. Another equal-time activity may answer the method-replacement question more directly.
The package adds time and practice as well as a named method. A relevant equal-time alternative asks whether replacing another activity with this method helps, rather than whether offering extra practice helps.
What to carry forward
- Every effect needs an outcome and comparator.
- Preserve units, denominator, direction and time.
- A difference becomes a causal estimate only under defensible design assumptions.
References
Sources supporting the explanations and learning checks in this lesson.