Learning Science · Effects, uncertainty and context

Replication and synthesis across studies

Build an interpretation from repeated tests and a structured comparison of studies.

Use the examples to connect this lesson with the earlier topics. No specialist equipment is needed.

After reading, you can

Distinguish reproducing an analysis from collecting new data.
Explain why counting positive studies is weak synthesis.
Identify the role of differences between studies.

Two ways to check a result

Reproducing an analysis asks whether the same data and specified methods yield the reported result. Replicating a finding asks whether new data addressing the same scientific question provide consistent evidence. Terminology varies across fields, so state the operation rather than relying only on the label.

Reproduction can reveal a computational or reporting problem. Replication adds evidence about whether the finding depends on a particular sample or arrangement. Neither requires every numerical estimate to be identical.

Specify what is being repeated

A close replication preserves important features to test whether a result recurs under similar conditions. A changed task or setting can test a proposed boundary or generalisation. If several features change at once, an inconsistent result may be difficult to explain.

An informative report describes the materials, participants, procedure and outcome in enough detail to compare studies. A failure to detect an effect can reflect imprecision as well as a smaller or absent effect. Read estimates and uncertainty together.

Synthesis is more than voting

Counting how many studies cross a p-value threshold ignores their sizes, precision, design quality and effect magnitudes. A systematic review defines a question, searches and selects evidence transparently, and evaluates the included studies. A meta-analysis may combine compatible numerical estimates.

Pooling is not automatically sensible. Studies might compare different alternatives or assess different capabilities. A combined number can conceal that mismatch. Explain which results answer the same question and which should remain separate.

Investigate differences without inventing a story

Heterogeneity can reflect participants, implementation, measurement, bias or genuine variation in effects. A review should examine those possibilities and the uncertainty around them. Finding a narrative after the fact is not the same as testing it.

Use accumulated evidence to state a revisable conclusion: what appears reasonably supported, under which conditions and where the evidence is thin. Repeated transparent tests can strengthen an explanation while preserving useful limits.

These are constructed study-design sketches, with no numerical results. Each includes retrieval practice.
Comparator	Practical question	Check before combining
No additional study	Does adding retrieval practice help?	Extra time and opportunity
Equal-time restudy	Does retrieval replace restudy beneficially?	Time, materials and outcome alignment
Guided explanation	How do these active approaches compare?	Feedback and explanation demands

Worked example

Three studies of the same activity

Three retrieval studies use different comparators. What should be organised before calculating one overall estimate?

Imagine three studies of retrieval practice: one compares it with no study, one with equal-time restudy and one with guided explanation. Their labels match, but their comparisons differ.
A review groups results by comparator and outcome before considering pooling. It also records delayed intervals and feedback. That organisation may explain why an overall slogan is less useful than a conditional account.

Group the actual comparator, outcome, interval and feedback conditions, then decide which effects answer a sufficiently similar question. A shared method label is insufficient.

Try it yourself

Think through each question before opening the answer.

Why is “seven positive studies and three negative studies” incomplete?
Show the answer and reasoning
The counts ignore effect magnitude, uncertainty, sample size, comparator, outcome and risk of bias.
A tally gives a very imprecise study the same vote as a precise one and erases effect size, comparator and possible bias. Synthesis needs the actual estimates and their meaning.
What does successfully rerunning the original code establish?
Show the answer and reasoning
That the analysis can be reproduced with those data and methods. It does not supply independent new observations or prove the causal interpretation.
Running the same code on the same data checks computational reproducibility. Replication collects new data, and neither activity automatically repairs confounding in the original design.

What to carry forward

Describe the operation meant by reproduction or replication.
Compare study questions before combining numbers.
Treat differences as research questions with uncertainty.

References

Sources supporting the explanations and learning checks in this lesson.