Learning Science · Evaluating emerging learning tools

Adaptive support and evidence of benefit

Evaluate what an adaptive system changes and whether that change supports the intended learning.

Use the examples to connect this lesson with the earlier topics. No specialist equipment is needed.

After reading, you can

Distinguish a system estimate from an observed capability.
Explain how task selection can help or mislead.
Design checks beyond the adaptive activity itself.

Adaptation is a decision process

An adaptive practice system chooses the next task using information such as previous answers. It may select a simpler example, offer a hint or return to an earlier topic. The system’s estimate is a model based on available evidence, not direct access to everything the learner knows.

A wrong answer can have several causes: an unfamiliar concept, misread wording, a motor error or an ambiguous item. If the model treats all errors alike, its next task may miss the actual need.

Inspect the learning opportunity

Ask what response the system observes and what it changes. Does it provide a useful explanation, merely repeat the same item or quietly avoid difficult content? A learner can experience many easy successes while receiving too little practice on the target capability.

Adaptation can also be valuable when it reduces an irrelevant barrier or offers prerequisite support. Judge the decision by its fit to the goal and actual response, rather than assuming that personalisation is inherently effective.

Separate prediction from benefit

A system may predict which item a learner will answer correctly without improving later learning. Prediction accuracy and instructional usefulness are different outcomes. To evaluate benefit, compare learning under the adaptive arrangement with a relevant alternative.

Keep the content, time, support and assessment clear. A favourable result on the same items used to tune a system may not establish performance on new items. Independent and delayed checks help test the claimed capability.

Each system claim asks a different question about an adaptive arrangement.
Claim	Relevant evidence	Insufficient substitute
Predicts a response	Accuracy on appropriate independent responses	A high activity score alone
Identifies the learning need	Reasoning and checks that distinguish possible error causes	Treating every wrong answer as the same deficit
Improves learning	Relevant later outcome against a feasible alternative	Prediction accuracy or easy-item success alone

Preserve a route to inspect and revise

A useful system lets learners or teachers notice when the diagnosis seems wrong. Show what the learner is practising, allow appropriate help and preserve access to explanations. An inaccurate model should be correctable rather than becoming a permanent label.

A local evaluation can examine both outcomes and experience: whether the tasks remain accessible, whether important content is omitted and whether learners can use the knowledge away from the platform. Version and implementation changes may alter the arrangement being evaluated.

Worked example

An error with two possible causes

A system treats a hard-to-see decimal point as failure to understand division. What evidence should change its next decision?

A learner selects the wrong unit-price answer because a small decimal point is hard to see. A system infers that division is not understood and assigns more division drills. The inference does not fit the cause.
A readable presentation and an explanation of the learner’s choice reveal the issue. The next check uses a fresh item with clear notation. Adaptation becomes useful only when its decisions respond to meaningful evidence.

Use readable notation and ask the learner to explain the choice, then check a fresh clear item. A system diagnosis should respond to the relevant cause rather than permanently label the learner.

Try it yourself

Think through each question before opening the answer.

Does accurate prediction of quiz answers prove improved learning?
Show the answer and reasoning
No. Prediction and learning benefit are separate claims. A comparison on relevant later outcomes is needed.
Correct prediction describes how well a model anticipates responses. Benefit asks what changes because the instructional arrangement is used; it needs a relevant comparison on the intended later capability.
Why inspect the tasks that an adaptive system stops assigning?
Show the answer and reasoning
It may be avoiding a needed capability rather than helping the learner develop it. Coverage matters alongside success rates.
A high success rate can be produced by selecting only easy content. Observing omitted tasks reveals whether the target remains practised rather than silently abandoned.

What to carry forward

An adaptive estimate is a revisable model.
Inspect the opportunities created by task selection.
Evaluate later learning with a relevant alternative and independent tasks.

References

Sources supporting the explanations and learning checks in this lesson.