Cosmology · Simulations, surveys and validation

Validation with independent observations

Compare two fixed models on held-out responses while retaining their exact common-error law and the limits of a covariance mismatch score.

Preparation: ratios, powers of ten and ordinary arithmetic. Physical quantities and notation are introduced as they are needed.

Back to the chapter

After reading, you can

Separate a calibration target from a held-out test.
Retain shared uncertainty between distinct observables.
Distinguish a covariance diagnostic from a fully specified probability law and physical validation.

A matched target leaves additional predictions to examine

Calibration uses declared information to choose or adjust model inputs. Agreement with that information is not automatically a prediction of the same strength as agreement with information held out of the adjustment.

Verification checks implementation of the stated equations. Physical validation evaluates a model against the relevant real system for an intended use. Calibration, numerical convergence and code comparison contribute different evidence.

Independent observations also need an uncertainty and response model. Distinct measured quantities can share calibration, physical assumptions or error sources, even when they have different parameter sensitivities.

Use a supplied dimensionless experiment to make those distinctions explicit. Its held-out values are synthetic and do not validate a cosmological theory against real observations.

Two parameter choices match the same calibration

Supply an exact calibration C = θ+β = 10. Fix model A at (θ,β) = (6,4) and model B at (8,2) before the held-out comparison.

Let the held-out responses be V₁ = 2θ+β and V₂ = θ+3β. A predicts (16,18), while B predicts (18,14).

Supply the synthetic measured vector d = (16,19). Define residual r as d minus the model prediction, giving r_A = (0,1) and r_B = (−2,5).

Neither model is adjusted using d in this comparison. The exact calibration fixes the sum θ+β, not θ and β separately, and both selected pairs satisfy that same target.

Fixed synthetic models; every quantity is dimensionless and residual means supplied d minus prediction
Model	θ	β	Calibration C	Prediction (V₁,V₂)	Residual	Full-covariance Q	Diagonal-only score
A	6	4	10	(16,18)	(0,1)	8/3	2
B	8	2	10	(18,14)	(−2,5)	104	58

Different responses can share one error contribution

Let held-out measurement errors be (S+e₁,S+e₂). Independently choose each of S,e₁,e₂ to be −1/2 or +1/2 with equal probability. These errors affect the held-out measurements, not the supplied exact calibration.

There are eight equiprobable underlying sign triples but seven distinct output error pairs. Two different triples give (0,0), so that pair has probability 1/4.

Each held-out error has variance 1/2 and their covariance is Var(S) = 1/4. The covariance matrix is C_V = [[1/2,1/4],[1/4,1/2]].

A covariance matrix alone would not specify this discrete law. Here the entire finite support and its masses are supplied explicitly, with no Gaussian assumption or continuous probability ellipse.

Complete distinct output error pairs; the zero pair has two of the eight underlying sign states
Error pair	Probability
(−1,−1)	1/8
(−1,0)	1/8
(0,−1)	1/8
(0,0)	1/4
(0,1)	1/8
(1,0)	1/8
(1,1)	1/8

Optional mathematical path: a quadratic score is not the likelihood

The inverse covariance is C_V⁻¹ = [[8/3,−4/3],[−4/3,8/3]]. Define the dimensionless mismatch diagnostic Q = rᵀC_V⁻¹r = (8/3)(r₁²−r₁r₂+r₂²).

This gives Q_A = 8/3 and Q_B = 104. Incorrectly discarding the off-diagonal covariance would give scores 2 and 58 instead.

The equivalent sum-and-difference expression is Q = (r₁+r₂)²/(3/2)+(r₁−r₂)²/(1/2). It retains the different shared and difference error directions.

Under the actual supplied error law, Q takes only 0 or 8/3 on its support. It has no automatic Gaussian chi-square probability calibration. The numerical score alone is not the full comparison.

Held-out evidence remains conditional on the tested model

Residual (0,1) occurs in the declared error support. Residual (−2,5) does not: each component error is bounded between −1 and 1, so B is incompatible with the supplied exact error mechanism for this d.

That support check uses information beyond covariance. Interpreting Q_B as an asymptotic significance or converting it into model odds would introduce assumptions that were not supplied.

Even a held-out comparison can share observational or theoretical dependencies with calibration work. Its scientific value requires checking those connections and the domain in which the model is intended to apply.

This construction shows how to separate a matched calibration, a new response test and their uncertainty assumptions. Actual physical validation needs relevant observations and an adequate response model; the synthetic experiment establishes neither a cosmic discovery nor a preferred cosmological mechanism.

Two supplied parameter pairs match the same exact calibration C = theta plus beta = 10: A is (6,4), B is (8,2). Their held-out predictions are (16,18) and (18,14). For supplied d = (16,19), the residuals are A (0,1) and B (minus 2,5). A full equal-unit coordinate plot shows all seven possible error pairs from independent S,e1,e2 each equal to plus or minus one half. Pairs (minus 1,minus 1), (minus 1,0), (0,minus 1), (0,1), (1,0), (1,1) each have probability one eighth; (0,0) has probability one quarter from two input states. A is exactly one allowed pair; B is outside the finite support. The covariance is [[1/2,1/4],[1/4,1/2]]. Full-covariance diagnostics are Q_A = 8/3 and Q_B = 104, while discarding covariance gives 2 and 58. The plot contains discrete probability masses rather than a continuous density or Gaussian confidence ellipse. Q is not automatically a significance or a likelihood, and the experiment is synthetic. Fixed synthetic models; every quantity is dimensionless and residual means supplied d minus prediction. Columns: Model; θ; β; Calibration C; Prediction (V₁,V₂); Residual; Full-covariance Q; Diagonal-only score. Rows: A; 6; 4; 10; (16,18); (0,1); 8/3; 2 / B; 8; 2; 10; (18,14); (−2,5); 104; 58. Complete distinct output error pairs; the zero pair has two of the eight underlying sign states. Columns: Error pair; Probability. Rows: (−1,−1); 1/8 / (−1,0); 1/8 / (0,−1); 1/8 / (0,0); 1/4 / (0,1); 1/8 / (1,0); 1/8 / (1,1); 1/8.

Worked reasoning

Retain both the shared error and its support

What does the supplied held-out comparison show about A and B?

Both models match exact C = 10, but predict different held-out responses.
With d = (16,19), the residuals are (0,1) for A and (−2,5) for B; full-covariance scores are 8/3 and 104.
The declared finite law allows (0,1) and excludes (−2,5); this support statement is separate from any Gaussian significance claim.

The models differ on held-out responses under an explicit shared-error law; the result remains conditional on that synthetic experiment.

Try the reasoning

Does fitting C = 10 determine θ and β separately?
Show the answer and reasoning
No.
The calibration fixes their sum; both supplied parameter pairs satisfy it.
Why are the held-out errors correlated despite different response equations?
Show the answer and reasoning
Both include the same random S.
Its variance 1/4 is the off-diagonal covariance.
Can Q_A = 8/3 be read directly as a Gaussian chi-square p-value?
Show the answer and reasoning
No.
The complete experiment has a finite eight-state input law; a covariance-weighted score does not supply that probability calibration.

What to carry forward

A calibration match and a held-out prediction test answer different questions.
Distinct observables may share error sources and modeling assumptions.
A covariance mismatch diagnostic is incomplete without the stated sampling law and validation domain.

References

Sources supporting the explanations, physical distinctions and examples in this lesson.