Reproducibility, open methods, literature evaluation and collaboration

State what was reproduced

This lesson uses computational reproducibility for obtaining consistent results from the same input data, methods and analysis conditions. It uses replicability for consistency across studies answering the same question with their own data. Other communities use these words differently, so the convention and the actual work should be stated.

Recomputing 4.5 from the retained four eligible values checks the calculation and inclusion rule. Recomputing 3.6 after replacing the missing record with zero instead follows a different analysis. A difference need not be a failure of the original arithmetic; the two readers may have used different inputs or rules.

The new observations five, seven, nine and one have mean 5.5. They are new evidence in this imaginary example. Whether real studies support consistent conclusions depends on design, population, uncertainty and context, rather than exact equality of two sample means. Successful same-data computation also cannot repair a biased original sample.

Preserve chronology instead of rewriting the plan

A durable plan records the question, intended comparison and relevant analysis decisions before their outcomes are known. Prespecification helps a reader distinguish planned analysis from exploration prompted by a pattern in the results. It does not prevent useful discovery or ensure that every planned method was sensible.

The original timeline records four planned contrasts, then the observations, then a fifth contrast suggested by those observations. The fifth can be reported as exploratory. A documented amendment identifies when a rule changed, why and what information was already known. It does not move a result-driven choice backward in time.

Conditional analysis can be planned: a stated decision rule may select an appropriate method if a specified condition is met. That differs from choosing among results because one looks more convincing. Openly reporting justified deviations and exploratory analyses gives later readers a clearer basis for evaluating the claims.

Evaluate a claim through its evidence and scope

A paper's title and reputation do not reveal how its measurements were obtained. Ask what question it actually addresses, which units were selected or assigned, how outcomes and exclusions were defined, and what assumptions connect its analysis to the conclusion. A review, an original study, an abstract and a protocol supply different kinds of information.

An abstract can establish that the authors report a finding and describe its stated scope, but it often cannot resolve all methodological details. A figure needs its definitions and underlying comparison; a numerical claim may require the original data or methods. Studies that repeat the same dataset do not supply independent new observations just because they appear in separate papers.

Collaboration also needs clear contributions and responsibilities. Records of who supplied data, developed an analysis, checked results and wrote an interpretation make the research easier to understand and correct. Appropriate credit, disclosed conflicts and respectful confidential review support that shared work; neither authorship nor independent review should be invented to fill a checklist.

Worked example

Check your understanding

1. What does an exact same-data recomputation establish in this example?

   Show reasoning

   It establishes that the named inputs and inclusion rule yield the reported summary again. It does not establish representative sampling, unbiased measurement or a valid causal conclusion.

2. May a result-driven fifth contrast be reported after four contrasts were planned?

   Show reasoning

   Yes, with its exploratory status and timing clear. It should not be presented as if it had been prespecified before the observations were known.

3. Why is one paper's abstract insufficient for every claim about its study design?

   Show reasoning

   The abstract reports a compressed account. Details about unit allocation, missingness, exclusions or analysis may require methods and underlying records. State the actual inspection extent rather than claiming a full review.

What to carry forward

Make the evidence relation explicit: same-data recomputation, new-data consistency and wider generalisation are separate claims.