Learning Science · Effects, uncertainty and context

Publication, measurement and interpretation biases

Recognise how missing results and flexible interpretation can distort the available evidence.

Use the examples to connect this lesson with the earlier topics. No specialist equipment is needed.

After reading, you can

Explain selective publication and selective outcome reporting.
Identify why a convincing story can emerge from many analyses.
Use transparency to assess a claim without treating it as a guarantee.

The visible studies may be a selected subset

Suppose researchers run ten studies, but only the most striking two are written up. A reader sees an unrepresentative view of the research effort. Publication bias concerns how the availability of studies relates to their results or other features. Missing evidence can distort a synthesis even when every visible calculation is correct.

Selective reporting can occur inside a published study. If twenty outcomes were measured and only the favourable one is highlighted, the report hides part of the comparison.

Flexibility can create an attractive pattern

Choices about exclusions, scoring, time points and subgroups can change results. Some choices are defensible and necessary. The problem is concealing their dependence on the observed outcomes or presenting a discovered pattern as an advance prediction.

Exploration is valuable for finding new questions. Its evidential role differs from a clearly specified prediction tested with data that did not generate it. Preserve that distinction rather than pretending research never needs to adapt.

A hypothetical study measures recall, explanation, speed and satisfaction at two times, but its favourable headline uses immediate satisfaction alone.
Statement	What it refers to	What would be needed
Immediate satisfaction favoured A	One reported feeling outcome at one time	Full reporting of the other planned results
A improves later explanation	A distinct performance and interval	The actual later-explanation result
A improves learning generally	A much broader claim	Appropriate outcome breadth, design and synthesis

Check the trail of decisions

A dated plan, accessible materials, an outcome list and an explanation of deviations help readers understand how the conclusion was reached. Registration alone does not validate the theory, measurement or analysis. A transparent weak design remains weak, but its limitations become easier to inspect.

Look for missing outcomes and participant flow as well as the main headline. If a result depends strongly on one reasonable analysis choice, report that sensitivity. Avoid selecting the only version that flatters the preferred explanation.

Do not diagnose bias from a single shape

A plot or a pattern among small studies may suggest missing evidence, but several processes can create similar appearances. Bias assessments require context and methods suited to the evidence set. They are not mind-reading tools for deciding whether a researcher intended to mislead.

A careful conclusion can say that the available evidence is uncertain because relevant results may be missing. That is different from asserting that the intervention definitely fails or that every positive finding is false.

Worked example

A flexible outcome becomes a fixed headline

A study measures four outcomes twice but highlights only favourable immediate satisfaction. What should its learning headline say?

A hypothetical study measures recall, explanation, speed and satisfaction at two time points. Only satisfaction immediately after the lesson is clearly favourable. A headline saying “proves improved learning” changes both the selected outcome and the scope.
A transparent report presents all planned outcomes, explains any additions and identifies the favourable satisfaction result as satisfaction. The next study can test a specific learning prediction.

Report that immediate satisfaction was the favourable result and preserve the other planned outcomes and times. A selected feeling measure cannot establish a general learning benefit.

Try it yourself

Think through each question before opening the answer.

Does preregistration prove a result is correct?
Show the answer and reasoning
No. It improves visibility of advance plans and deviations, but design, measurement, execution and analysis still need evaluation.
Registration can expose plans and changes; it does not validate the measure, remove confounds or guarantee implementation. Quality still depends on the design and the evidence actually collected.
Why can a meta-analysis be biased even if its arithmetic is flawless?
Show the answer and reasoning
Its available studies or selected outcomes may be incomplete in ways related to results. Correct pooling cannot recover evidence that is systematically absent.
A mathematically correct combination can summarise a distorted visible set. Searching for missing studies and results addresses information the pooling formula cannot restore.

What to carry forward

Ask what evidence and outcomes are missing.
Separate discovery from advance prediction.
Use transparency to inspect choices, not as automatic approval.

References

Sources supporting the explanations and learning checks in this lesson.