Learning Science · What does a learning measure measure?
Validity, reliability and measurement conditions
Separate consistency of measurement from justification for a particular score interpretation.
Use the examples to connect this lesson with the earlier topics. No specialist equipment is needed.
Back to the chapterAfter reading, you can
- Explain why consistent scores can support an inappropriate conclusion.
- Identify sources of score variation.
- Match measurement precision to the intended decision.
Two questions about the same score
Imagine a ruler whose marks are evenly spaced but begin two centimetres away from its edge. Repeatedly aligning the edge with a small object gives consistent readings with a systematic offset. Consistency and appropriateness are different questions. Educational measurement has a similar distinction: reliability concerns dependable scores under specified repetitions, while validity concerns evidence for an interpretation and use.
A vocabulary quiz might consistently rank learners while providing weak evidence about their ability to hold a conversation. Reliability does not repair a mismatch between the task and the capability.
Specify what would be repeated
Would the same learner take another set of questions, respond on another day or receive a score from a different assessor? Each repetition exposes different possible errors. A long essay might be stable across topics but scored differently by two readers. A tightly scored quiz might vary because its small sample happens to include unfamiliar terms.
Precision is a property of scores in a population and measurement arrangement. A published coefficient from one population does not automatically describe a local classroom, changed language or new scoring rule.
| What is repeated | What changes | Question it can inform |
|---|---|---|
| Same response scored again | Assessor or scoring occasion | Are criteria applied consistently? |
| New item sample | Questions within the intended domain | Does the score depend strongly on selected items? |
| Later occasion | Time and testing conditions | Is performance dependable under that arrangement? |
Build an interpretation argument
Suppose the claim is that a learner can explain proportional relationships. Inspect the task coverage, the reasoning learners actually use, the scoring criteria and performance on other relevant tasks. A clear rubric and examples can help scorers distinguish a sound relationship from a correct number reached accidentally.
Evidence is needed for the proposed use. Choosing the next practice exercise demands a different level of support from using one score to make a major placement decision. More consequential decisions need stronger justification and appropriate professional oversight.
Use variation as information
Record the conditions and examine patterns rather than treating a single difference as a precise change in capability. A learner whose scores fluctuate might be encountering different content, unclear instructions, fatigue or inconsistent scoring. Repeating exactly the same exposed answers can increase familiarity without improving the measurement of broader knowledge.
For everyday teaching, retain representative work and the reasons for feedback. If a narrow score is uncertain, a focused conversation or additional task may resolve the next instructional step better than adding decimal places.
Worked example
A rubric for explanations
Two scorers disagree because one rewards the word “heat” and the other expects a mechanism of evaporation. What needs aligning, and what would agreement still leave open?
- Two scorers judge the same five explanations of evaporation. One awards credit whenever the word “heat” appears; the other requires an account of particles leaving the surface. Their disagreement concerns the criterion, not simply careless arithmetic.
- They agree on the intended explanation, compare examples and score fresh responses independently. Greater agreement is useful evidence about scoring consistency. They still need to check whether the task elicits the intended explanation and covers enough relevant situations.
Align the intended criterion and practise scoring fresh responses independently. Agreement concerns scoring consistency; it does not alone establish coverage or validity for the proposed use.
Try it yourself
Think through each question before opening the answer.
Can a highly consistent multiple-choice quiz prove practical competence?
Show the answer and reasoning
Only if evidence supports that interpretation. Consistency within the quiz does not demonstrate performance of the practical task.
Consistent quiz scores concern the repeated measurement arrangement. Practical competence requires relevant work and evidence that the proposed interpretation is justified.
Why is a score change from 7 to 8 out of 10 uncertain evidence of progress?
Show the answer and reasoning
The items, conditions and scoring may contribute variation. Inspect the actual responses and repeat a relevant check before interpreting the one-point difference as a durable gain.
The one-point difference may involve different item content, support or scoring. The responses and their conditions reveal more than treating the score as exact change.
What to carry forward
- Consistency and validity answer different questions.
- Specify the population, task, scoring and intended use.
- Inspect the reasoning behind a score before making a large claim.
References
Sources supporting the explanations and learning checks in this lesson.