Learning Science · Effects, uncertainty and context
Variation across learners, tasks and settings
Ask whether differences in an effect are supported by a direct comparison and a plausible design.
Use the examples to connect this lesson with the earlier topics. No specialist equipment is needed.
Back to the chapterAfter reading, you can
- Distinguish outcome variation from variation in an intervention effect.
- Explain what a moderator claim requires.
- Avoid treating a group average as an individual rule.
Learners differ for many reasons
After the same lesson, some learners score highly and others struggle. That is variation in outcomes. It does not by itself show that the lesson had a different effect on each learner. Learners may have begun at different points or encountered different conditions. An effect concerns a comparison with an alternative, which must also be considered.
To investigate whether a method helps novices more than experienced learners, compare methods within the relevant groups and then examine how those comparisons differ.
A moderator changes the comparison
A moderator is a characteristic associated with variation in the estimated relationship or effect. Prior knowledge is a plausible candidate when the instructional support assumes certain knowledge. The claim still needs an appropriate analysis and uncertainty, not just a story that sounds sensible.
If a result is statistically significant in one group and not significant in another, that does not establish that the effects differ. The second group may simply have fewer observations or more variable scores. The difference between effects must be examined directly.
| Group or contrast | Point estimate | Permitted reading |
|---|---|---|
| Beginners: A minus B | +3 points | Estimated within-group method difference |
| Experienced: A minus B | +2 points | Estimated within-group method difference |
| Difference between those effects | 3 − 2 = 1 point | Needs its own uncertainty analysis |
Beware of searching until a pattern appears
A dataset can be divided by age, prior score, task, session, interest and many other features. Inspecting enough divisions can produce striking patterns by chance. Prespecifying a small set of meaningful comparisons and reporting all of them makes interpretation clearer.
Exploratory patterns can be useful leads. Label them as exploratory and test them with suitable new evidence. They should not immediately become a fixed instruction for everyone who shares the selected label.
Use variation to improve questions
If an explanation seems useful only when learners know a prerequisite, inspect that prerequisite and the teaching mechanism. Perhaps the next study should compare added preparation rather than assign learners to permanent types.
Within a classroom, respond to actual work. A group average may guide the choice of examples, but an individual’s questions and responses are better evidence for that person’s next step. Support can be adapted without claiming a scientifically validated personal learning profile.
Worked example
The misleading two-group headline
Does a statistically significant advantage of +3 for beginners and a nonsignificant +2 for experienced learners show that only beginners benefit?
- In a constructed study, method A has an estimated advantage of three points for beginners and two points for experienced learners. Only the beginner estimate passes a significance threshold. “Works only for beginners” does not follow.
- The estimates are close, and their uncertainties may overlap substantially. A direct test of their difference, planned in the design, is needed to support the moderator claim.
No. The point-estimate difference is one point. Whether effects differ needs a direct comparison with uncertainty; two separate significance labels do not establish that difference.
Try it yourself
Think through each question before opening the answer.
Why do different final scores not prove different treatment effects?
Show the answer and reasoning
Starting knowledge and other conditions can produce score differences even if the intervention’s effect is similar. An effect requires a comparison with an alternative.
Different final scores may reflect initial knowledge or other starting conditions. Variation in the treatment contrast is a different question from variation in observed outcomes.
What should happen to an unexpected subgroup pattern?
Show the answer and reasoning
Report how it was found, preserve the other analyses and seek an informative test with new or appropriately independent evidence.
An unexpected subgroup may be a useful question. Preserving the full analysis set and testing the idea with fresh evidence prevents a selected discovery from masquerading as an advance prediction.
What to carry forward
- Outcome differences and effect differences are distinct.
- Compare effects directly before claiming moderation.
- Exploratory patterns generate questions, not fixed learner categories.
References
Sources supporting the explanations and learning checks in this lesson.