Data versions, provenance and reproducible inference

A result depends on more than its data file

An inference combines recorded data with a response model, calibration, sampling assumptions, parameter priors and a calculation. Keeping only a final number leaves those dependencies unresolved.

Reproducibility asks whether a declared calculation can be replayed from its identified inputs. Scientific validation also asks whether those inputs, response and assumptions adequately describe the intended physical situation.

Versioned configurations and source history can reveal what changed between results. Checksums identify exact file bytes within that record; they do not establish the truth or completeness of the model.

Cosmological software workflows supply concrete examples of linked theory, observational likelihood and parameter configuration. Here we perform a small original enumeration without running a sampler or adopting observational data.

Three candidate amplitudes and a specified noise law

Choose arbitrary readout unit U and exactly three allowed amplitudes A/U ∈ {1,2,3}. The discrete prior is a supplied model assumption, not an approximation to an undeclared continuous prior.

Each run contains two ordered, conditionally independent readings rᵢ = A+b+eᵢ. The calibration b is a fixed versioned input, either 0 or U, rather than an unknown quantity integrated out.

For each reading, the independent error eᵢ/U has values −1,0,1 with probabilities 1/4,1/2,1/4. Residuals outside this support have zero probability. All rows are retained in their declared order.

For each candidate A, calculate L(A) as the product of the two residual probabilities. Then Z = Σ_A L(A)π(A), and posterior mass is L(A)π(A)/Z only when Z > 0.

Calibration, data and prior revisions have distinct effects

Run V1 supplies readings (2,3)U, b = 0 and uniform prior masses (1/3,1/3,1/3). Its likelihoods at A/U = 1,2,3 are (0,1/8,1/8).

Run V2 uses the same raw reading bytes and prior, changing only b to U. The likelihoods become (1/8,1/8,0), so the posterior support shifts to smaller amplitudes.

Run V3 returns to b = 0 and the uniform prior, changing only the V1 readings to (2,2)U. Run V4 keeps V3’s readings and calibration but changes prior masses to (1/2,1/4,1/4).

These are explicit synthetic revisions. A fixed calibration change is not marginalization over calibration uncertainty, and changing a prior is not changing the raw observation.

Preserve exact inputs and their meaning

V1 and V2 have matching raw reading-file checksums, yet posterior means 2.5U and 1.5U. The calibration configuration changed, so data identity alone does not determine the output.

V3 and V4 also share reading bytes and the same likelihood values. Their different prior masses change the normalization and posterior mean from 2U to 13U/7.

A complete replay record identifies raw data, ordering and selection, units, calibration, candidate space, sampling law, prior, calculation and relevant program or runtime versions. Exact byte checksums complement this semantic record.

This finite enumeration has no random seed, Markov chain or Monte Carlo convergence claim. Its arithmetic can be checked exhaustively, including the nine possible ordered reading pairs for each fixed candidate and calibration.

Reproducing a calculation does not validate its assumptions

If the supplied readings were (100,100)U with b = 0, every allowed candidate would require errors outside the declared support. All likelihoods and the total evidence would be zero.

No normalized posterior exists in that condition. The appropriate result is an incompatible-data/model-condition report, followed by investigation of units, inputs or assumptions; dividing by zero or inventing posterior masses would conceal the failure.

A replay can reproduce a flawed calibration or inadequate sampling law perfectly. Assertions, independent numerical checks and comparison with relevant physical observations address different parts of validation.

For cosmological inference, preserve provenance and ask which changed dependency moved the result. An exact replay is valuable evidence about the calculation, while adequacy of the physical and observational model remains a separate scientific question.

Worked reasoning

Try the reasoning

1. What should be reported when every candidate gives zero probability to the data?

   Show the answer and reasoning

   Zero evidence and incompatible declared conditions; no normalized posterior.

   The normalizing denominator is zero, so posterior masses cannot be defined by this model.

2. Why do V3 and V4 differ despite the same likelihood values?

   Show the answer and reasoning

   Their prior masses differ.

   Multiplying by the new prior gives evidence 7/64 and posterior (2/7,4/7,1/7).

3. Does reproducing the same output establish that the noise law describes real observations?

   Show the answer and reasoning

   No.

   Replay verifies a declared computation; the observational adequacy of its assumptions needs separate evidence.

What to carry forward

• An inference depends on identified data, response, calibration, noise, prior and calculation.
• Exact finite replay can isolate the effect of one changed input.
• Zero evidence must remain visible, and reproducibility does not establish scientific validity.