Astronomy · Testing models against the sky

Where a model fails and how it is revised

Inspect an exact residual pattern that survives a high fit score, then compare a bounded quadratic revision and preserve the independent role of a new test.

Continue in the approved chapter order. Necessary vocabulary, quantities and assumptions are explained in the lesson.

Back to the chapter

After reading, you can

Distinguish implementation, numerical, calibration and predictive failures before choosing a revision.
Calculate residual structure and least-squares constraints for a synthetic line fit.
State the evidence, uncertainty and intended-use domain needed for a revised scientific claim.

A mismatch is a question to investigate

A failed comparison can be valuable scientific information, but its cause is not written into the residual. A program may implement the wrong equation, a numerical solution may be insufficiently converged, a reference may be mismatched, or a physical model may omit a relevant effect. These possibilities call for different checks.

Ask first which claim failed. Does the implementation reproduce an exact test problem? Does the relevant observable converge with numerical refinement? Do independent calibration checks agree in the operating range? Does the physical prediction match an observation outside the fit, with its uncertainty and selection properly represented?

Preserve the version, inputs and conditions that produced the failure. A later correction should not erase the record of what the earlier model predicted. That history makes it possible to distinguish repairing a calculation from revising an astronomical explanation.

The following original response example has no observational noise. It isolates a missing functional shape so that its arithmetic can be checked exactly. It is not evidence that a real astronomical source has been discovered to follow a quadratic law.

Inspect the errors that a scalar metric compresses

Let u be a dimensionless explanatory quantity and assign the normalized response y = 10 + 2u + 0.5u² at u = −2, −1, 0, 1 and 2. The supplied values are 8, 8.5, 10, 12.5 and 16. Fit an unweighted least-squares line a + bu to those five targets.

The mean input is zero and the mean response is 11. The fitted slope is Σuyi/Σu² = 20/10 = 2, and the intercept is 11. The line therefore predicts 7, 9, 11, 13 and 15.

Define residual ei = yi − (11 + 2ui). The residual sequence is +1, −0.5, −1, −0.5, +1. It forms a symmetric curved pattern: the fitted line is low at both ends and high near the centre. That structure is visible in the complete table.

The sum of squared residuals is SSE = 1 + 0.25 + 1 + 0.25 + 1 = 3.5. The total sum of squared deviations from the response mean is 43.5, so R² = 1 − 3.5/43.5 = 80/87 ≈ 0.919540. That relatively high score has not removed the systematic missing-curvature pattern.

Original dimensionless response targets and unweighted line fit. Residuals are target minus prediction; no random measurement law is assigned.
u	Assigned response y	Line 11 + 2u	Line residual	Quadratic 10 + 2u + 0.5u²
−2	8	7	1	8
−1	8.5	9	−0.5	8.5
0	10	11	−1	10
1	12.5	13	−0.5	12.5
2	16	15	1	16

Zero sums do not establish independent noise

The residuals sum to zero: 1 − 0.5 − 1 − 0.5 + 1 = 0. Their input-weighted sum is also zero: (−2)(1) + (−1)(−0.5) + 0(−1) + 1(−0.5) + 2(1) = 0.

Those identities follow from the least-squares normal equations for a line with an intercept. The fitted residual vector is orthogonal to the constant and linear input columns. That algebraic condition can hold while a clear nonlinear pattern remains.

The descriptive root mean squared error is √(3.5/5) ≈ 0.836660 in normalized response units. It is not measurement uncertainty in this noise-free construction. No chi-squared significance or confidence interval follows without a suitable sampling model and uncertainty information.

NIST’s model-validation discussion emphasizes inspecting residual behaviour beyond a single R² value. In real data, look against relevant predictors, fitted values, observing order, location and conditions. A visually unstructured plot also does not prove every model assumption, especially when few residual degrees of freedom remain.

A better fit creates a new candidate with a new evidence boundary

Propose the quadratic response q(u) = 10 + 2u + 0.5u². It reproduces every supplied target and has zero fitted SSE in this example. This is expected because the targets were generated from that polynomial; it is not an independent empirical discovery.

Now supply an additional target at u = 3, y = 20.5, held out of the quadratic fit. The line predicts 17 and leaves residual 3.5. The quadratic predicts 20.5 and agrees exactly. The new point tests the proposed prediction outside the fitted input range [−2,2].

The agreement supports this bounded comparison with this stipulated point. It does not determine an arbitrary response at every larger u or prove a unique physical law. Other sufficiently flexible functions can fit finitely many targets too, as the earlier finite-node construction showed.

If this test result leads to another revision or candidate choice, record that use. It then contributes development information, and a fresh independent evaluation of the newly selected procedure needs new information. Keep the old prediction and its test outcome available.

Adding flexibility can conceal as well as explain a failure

A physical discrepancy term represents inadequacy of a model for the intended system. It differs from uncertain parameter values, repeated-measurement noise, numerical error and a shared reference bias. All can influence residuals, but giving them one unrestricted noise label loses the distinctions needed for prediction.

A freely adjustable discrepancy function can absorb almost any finite residual. If it also imitates a parameter change, the parameter and discrepancy may remain confounded. A perfect fit then need not identify either one or provide a defensible extrapolation interval.

For real astronomical data, inspect reduction, background, response, calibration and sample selection before attributing a pattern to new physics. Compare bounded alternatives and ask which new observation would make their predictions differ. Additional observing regimes can be useful when their references and uncertainty are independently constrained.

A revision should state what changed, why that change is scientifically plausible, which data informed it, what remains uncertain and the domain in which its use is supported. NASA’s modelling standard distinguishes that domain and version history; preserving them makes a model’s scientific limits inspectable.

A useful model states where its evidence reaches

Revising a model can mean correcting software, improving numerical control, adding an independently justified calibration effect, restricting the domain, or changing the physical explanation. Report the kind of revision actually made. A repaired implementation and a new physical theory should not be described with the same evidence claim.

A concise failure record includes the tested prediction and uncertainty, the observation or referent, the diagnostic result, candidate causes, the chosen revision and its new predictions. Preserve unsuccessful alternatives when they explain a remaining ambiguity or the limits of the test.

The chapter has separated mathematical accuracy, finite-design identification, held-out testing, learned prediction and model revision. Together they help turn a numerical fit into a bounded scientific argument without treating agreement as universal approval.

The following chapter connects astronomical observables with larger-scale questions. It begins with measuring redshift before interpreting it, while leaving the separate Cosmology course to develop its own physical models and parameter inferences.

Worked examples

Check the score and the pattern together

What R² follows from SSE = 3.5 and total response sum of squares 43.5, and does it remove the residual pattern?

Use R² = 1 − SSE/SST.
Compute 1 − 3.5/43.5 = 80/87 ≈ 0.919540.
Inspect the actual residuals +1, −0.5, −1, −0.5, +1.

R² ≈ 0.919540, while the curved residual pattern remains. The score alone does not certify adequacy.

Compare the two predictions at the new input

At the held-out u = 3 target y = 20.5, what do the line and revised quadratic predict?

The line gives 11 + 2 × 3 = 17, residual 3.5.
The quadratic gives 10 + 6 + 0.5 × 9 = 20.5, residual zero.
Keep this newly supplied point separate from the fitted five-row targets.

The quadratic matches this stipulated test point and the line does not. That result does not prove universal extrapolation or a unique physical mechanism.

Try the reasoning

Does a zero sum of fitted line residuals establish that they are independent random errors?
Show the answer and reasoning
No.
It is an algebraic least-squares constraint and can coexist with systematic curvature.
Is √(3.5/5) automatically a measurement uncertainty?
Show the answer and reasoning
No.
Here it is a descriptive deterministic fit error without a measurement-noise model.
Does a discrepancy function able to fit every residual necessarily identify the physical parameters?
Show the answer and reasoning
No.
It may mimic parameter changes and leave them confounded.
Does a failed physical comparison by itself show that new astrophysics is responsible?
Show the answer and reasoning
No.
Implementation, numerical, reference, reduction and model assumptions need distinct checks.

What to carry forward

Identify which claim failed before choosing a correction.
Residual structure can survive high fit scores and algebraic least-squares checks.
Preserve revisions, independent tests and the domain supported by their evidence.

References

Teaching resources and research supporting this lesson’s explanations and examples.