Calibration, archives and supporting infrastructure

Scientific capability includes the retained data chain

An instrument’s scientific value extends beyond its initial exposure. Archives allow observations to be recalibrated, compared and used for questions that were not the original target. That value depends on retaining enough information to interpret the data and its transformations.

Useful records include raw measurements where available, instrument configuration, observing conditions, units, coordinate and time conventions, calibration references, uncertainty and quality flags. A processed table without these relationships can be difficult to compare even if its numerical columns look precise.

Calibration relates indications to quantity values under specified conditions, with uncertainty. Different reference data or response models can change a derived product without changing the original photons. A reprocessed catalogue entry is therefore not automatically a new independent observation.

The Foundations accuracy, trueness and precision record distinguishes closeness to a reference from agreement among repeats. Small scatter or many printed digits cannot by itself establish small systematic error. The uncertainty budget needs the actual measurement model and relevant shared inputs.

A newer input must still be compatible with its processing

The STScI JWST calibration documentation describes pipeline steps that use instrument-specific reference files and configurations. Those files can include response corrections, masks or processing parameters. The resulting product belongs to the combination of observation, software, reference context and configuration that generated it.

The documentation also explicitly distinguishes compatible reference contexts from newer files that may not work properly with older pipeline versions. Newer does not mean automatically suitable for every software combination or science case. Preserve the version and context used when comparing products.

Reprocessing can improve a result by correcting a known problem or applying a better-supported response. It can also change selection, uncertainty or correlations. Compare the actual transformed quantities and their conditions rather than assuming every numerical difference is a change in the sky.

An archive should retain that lineage alongside the output. Provenance describes which entities and activities produced a dataset and who or what was responsible. With adequate detail, it helps trace a questionable input forward to affected products or a result backward toward its observations and calibration.

Large totals and small offsets need suitable storage

For an original numerical exercise, use one uniform coordinate in SI seconds with an exact reference origin. Let the exact base be T = 2²⁴ = 16777216 s, and supply offsets 0.25, 0.75, 1.25 and 1.75 s. No date, leap-second or astronomical reference-position conversion is being performed.

IEEE binary32 has 24 significand bits in its normal range. At and immediately above T, within [2²⁴, 2²⁵), its adjacent representable values are spaced by 2 seconds. The four exact totals lie between T and T + 2; assume rounding to nearest with ties to even. None of these inputs is a rounding tie.

The totals T + 0.25 and T + 0.75 round to T. The totals T + 1.25 and T + 1.75 round to T + 2. Subtracting the exact base from those stored totals recovers offsets (0, 0, 2, 2) seconds, rather than the four supplied values.

Representation errors, defined as stored total minus exact total, are respectively −0.25, −0.75, +0.75 and +0.25 s. These are deterministic storage effects in a stated format, not measured clock noise or evidence that a real observatory archive has this fault.

A suitable representation preserves these supplied offsets

The exact consecutive time differences are (0.50, 0.50, 0.50) s. Differences of the stored binary32 totals are instead (0, 2, 0) s. A later calculation can be internally correct on the stored values while answering a distorted version of the intended timing question.

One option for this exact exercise is to retain the integer base and the four small offsets as separate fields. These quarter-second offsets are exactly representable in binary32. Taking their differences without first collapsing the large totals gives the correct three half-second intervals.

Another option here is binary64, which has 53 significand bits for normal values. At and immediately above T its spacing is 2^(24 − 52) s = 2⁻²⁸ s, approximately 3.725290298 × 10⁻⁹ s. All four supplied totals are exactly representable in that format.

These options illustrate a representation choice, not a general guarantee of numerical accuracy. If the split fields are recombined into binary32 totals, the same loss returns. Different values, operations, overflow, underflow or accumulated rounding can require further analysis.

Exact arithmetic still needs physical calibration

A day-number format or an ISO-style string describes a representation. It does not by itself identify the physical time scale, origin, observatory reference location, clock correction or whether a timestamp marks exposure start, middle or end. Those conditions affect comparisons between observations.

Astropy’s time documentation distinguishes formats from scales and uses a two-double internal representation. That is an example of preserving numerical timing information while interpreting time conventions. The simple integer-plus-offset exercise above is not an implementation or benchmark of Astropy’s algorithms.

Representation precision, display precision and physical timing accuracy are separate. An exactly stored timestamp can still have uncertain synchronization or an incorrect reference correction. More displayed decimals do not repair either the physical calibration or information already discarded during storage.

Metrological traceability connects a particular measurement result to a stated reference through a documented calibration chain with uncertainty contributions. Data provenance records production history. Both are useful, but an archive address, checksum or instrument label alone does not establish that the measurement uncertainty is adequate for its intended use.

An archive supports tests when its dependencies remain visible

Retain compatible software and reference identities, configuration, transformations, quality decisions and the observation data needed for the intended reuse. Preserve units, frames, time scales and uncertainty alongside the values. Record what a correction changed and which products share the affected input.

A checksum helps identify exact retained bytes, while provenance helps explain what those bytes represent and how they were produced. Neither reconstructs an offset erased before storage or proves a physical interpretation correct. The evidence chain needs content, meaning and calibration together.

When two catalogues disagree, determine whether they use distinct observations or reprocess the same exposures, whether calibrations are shared and how the definitions and selections differ. Counting both products as independent photons would misrepresent the joint evidence.

Supporting infrastructure enables new science by preserving interpretable measurements and making revised comparisons possible. The final lesson brings the course together by distinguishing physical constraints, engineering requirements, demonstrations and uncertain scientific outcomes.

Worked examples

Try the reasoning

1. Can printing more decimal places recover the offsets lost from the binary32 totals?

   Show the answer and reasoning

   No.

   Formatting cannot restore information already discarded.

2. Why must the exact base and small offsets remain separate or be combined at adequate precision?

   Show the answer and reasoning

   Collapsing their large totals back to binary32 repeats the same rounding loss.

   The benefit depends on the actual storage and arithmetic operations.

3. Is a newer calibration reference automatically compatible with every older pipeline?

   Show the answer and reasoning

   No.

   The software and reference context must be checked as a combination.

4. Does exact numerical storage establish physical clock accuracy?

   Show the answer and reasoning

   No.

   Synchronization, time scale, reference position, exposure convention and calibration uncertainty remain relevant.

What to carry forward

• Calibration, compatible processing and provenance are parts of observing capability.
• Numerical representation can discard small differences before scientific analysis.
• Precise storage and a documented history do not replace physical calibration or fitness-for-purpose uncertainty.