Reproducible analysis, provenance and transparent reporting

Preserve the route from observations to a reported result

A reported astronomical result should be interpretable by someone who did not run the original analysis. They need to know which observations were used, which quantities were estimated, how instrumental effects were treated, what was excluded and what assumptions connect the record to the scientific conclusion.

Provenance is that account of origins and transformations. In the International Virtual Observatory Alliance provenance model, an entity can be an input or output dataset, an activity can be a processing step, and an agent can be a person or organization responsible for an activity. Relations identify which entity was used in which role and which output was generated.

Those distinctions are useful even for a small notebook without a formal provenance service. A raw exposure, a flat-field reference, a quality mask and a configuration are all inputs, but they have different scientific roles. Saving only their filenames without identifying their contents or roles may be insufficient when files change.

The goal is enough detail to interpret the stated result and attempt its reproduction. It is not a guarantee that every archived file is correct or that every environment will yield identical bytes. State the comparison being attempted and the conditions held fixed.

The extraction rule is part of the quantity

Consider three original synthetic pixels observed for 10 seconds. Each has a raw indication D in ADU, a bias B = 2 ADU, an accumulated dark contribution Q = 0 ADU and a dimensionless relative flat f. Define its corrected rate as r = (D − B − Q)/(ft), in reference-response ADU s⁻¹.

The input values are A: D = 12, f = 0.5, quality good; B: D = 20, f = 1, quality good; and C: D = 100, f = 1, quality hot. The quality flags are supplied externally and fixed for this example. They are not thresholds chosen after inspecting the resulting rates.

Define the reported quantity as the sum of rates over exactly the pixels flagged good in this specified region. The pixel rates are 2, 1.8 and 9.8 reference-response ADU s⁻¹. Only A and B enter the defined output, giving 3.8 reference-response ADU s⁻¹.

That is a retained two-pixel-region rate. It is not an estimate of omitted sky light, a missing-pixel correction or total source flux. Such quantities would need additional spatial, instrumental and source assumptions. Retaining the extraction rule prevents an apparently simple sum from acquiring an unintended physical meaning.

Record revisions rather than overwriting their history

Now change only the flat for pixel A from 0.5 to 0.4, retaining the same raw data, additive terms, duration and mask. Pixel A becomes (12 − 2)/(0.4 × 10) = 2.5, and the retained sum becomes 4.3 reference-response ADU s⁻¹. This is a different calibration result, even if both runs use a file called flat.fits.

Alternatively, keep the original flat and omit the fixed quality mask. The sum becomes 2 + 1.8 + 9.8 = 13.6 reference-response ADU s⁻¹. That is a different selection of pixels and a different defined output. A repeated execution of this altered procedure does not reproduce the original masked quantity.

Retain original and revised products with their input identities, processing version and change reason. A comparison can then attribute the 3.8-to-4.3 change to the flat rather than to new observations or a changed mask. If several inputs change together, the provenance should say so instead of implying that one isolated cause was tested.

The CRDS documentation provides a practical astronomical counterpart: calibration contexts can be specified rather than left to a changing default. A frozen context and the actual reference files help identify an analysis. The context name alone does not explain the extraction region, scientific model or every numerical dependency.

Identical files and agreeing scientific numbers are different checks

A content hash can establish that a retained artifact matches a particular byte sequence. It cannot establish that the flat was suitable, that a hot-pixel flag was correct, or that the scientific model was adequate. An incorrect mask can have a perfectly reproducible hash.

Bitwise computational reproduction asks for exactly identical output bytes under stated conditions. Numerical reproduction may instead ask whether designated values agree within a declared tolerance, with the same units and scientific definitions. Floating-point libraries, hardware, operation order or nondeterministic execution can affect exact bytes even when the selected numerical comparison agrees.

A software version label, a random seed or a saved script is useful but may leave input data, calibration references, configuration or dependencies unspecified. Record the relevant environment and precision along with the algorithm. If randomness is used, preserve its role and controls; do not assume that a seed freezes every other input.

Metrology uses repeatability and reproducibility for precision under specified measurement conditions, including which operators, locations and systems change. Our computational comparison is an explicitly stated operational task. It should not be silently substituted for those measurement-precision definitions or for independent replication of a physical observation.

Transparent reporting includes assumptions and exclusions

For the toy, a useful report identifies the raw values, bias and accumulated-dark references, flat values, exposure duration, quality flags, included pixels, correction order, units and output. For real work, add observation identifiers, software and calibration versions, relevant warnings, numerical conditions and access information sufficient to identify the actual inputs.

The scientific record also needs the sampling and selection rules, background treatment, physical and statistical model, fitted or fixed parameters, uncertainty components and correlations, and the meaning of any interval or upper limit. Reproducing a point estimate without its assumptions does not reproduce its interpretation.

Credit the data and software contributors and identify the versions used. The AAS software guidance distinguishes a publication describing software from a version-specific software citation. Both can help: one explains the method and gives credit; the other identifies the implementation behind this analysis.

Finally, preserve the limits of the comparison. Agreement with the original run supports faithful execution under the declared conditions. Independent calibration checks and tests of model adequacy support different conclusions and remain necessary where the scientific question requires them. The next chapter starts earlier in this chain: choosing observations whose possible outcomes can answer the intended question.

Worked examples

Try the reasoning

1. Does summing all three original pixel rates reproduce the defined masked result?

   Show the answer and reasoning

   No; it gives 13.6 rather than 3.8 reference-response ADU s⁻¹.

   Including the externally flagged hot pixel changes the extraction rule.

2. Does a matching file hash prove that a calibration is scientifically suitable?

   Show the answer and reasoning

   No.

   It establishes artifact identity, not suitability or correctness.

3. Is a software version alone enough to identify an analysis that can use changing calibration defaults?

   Show the answer and reasoning

   No.

   The actual calibration context, references, input data and settings also matter.

4. If exact bytes differ but designated values agree within a predeclared tolerance, what should be reported?

   Show the answer and reasoning

   Numerical agreement under that tolerance, while noting the failure of byte identity.

   The two comparisons test different stated criteria.

What to carry forward

• Provenance records input roles, processing choices and the outputs they produced.
• A changed calibration or mask creates a distinct analysis result whose history should remain inspectable.
• Reproduction and artifact identity support traceability; scientific validity also requires appropriate calibration, uncertainty and model checks.