Independent tests and held-out observations

A filename does not make a test independent

Data used to fit parameters help construct a model. Data used to choose features, penalties, architectures or alternative models also help construct the selected procedure. A final held-out evaluation asks how that selected procedure behaves on information not used to make those choices.

Machine-learning practice often calls the tuning subset validation data. That specific development role should be distinguished from the broader scientific question of whether a model represents a system adequately for its intended use. A low tuning error does not by itself answer that broader question.

If a supposedly final test is inspected repeatedly to choose the best model, it has become development information. Renaming it test.csv or keeping its rows outside the original training file does not undo that use. Preserve the history of access and decisions.

An astronomical evaluation also needs a target: new sources from the same population, a different instrument, a future observing period or an extrapolation to another physical regime. A test can be independent in one sense yet fail to represent the intended new use.

Use a complete ledger rather than one selected score

Continue the preceding lesson’s dimensionless candidates A(u) = 1 + u² and B(u) = 1 + u² + 0.25u(u − 1)(u − 2). Supply training/calibration targets (u,y) = (0,1), (1,2) and (2,5). Both candidates reproduce all three exactly.

For a declared synthetic development protocol, next reveal two inside-range tuning targets: (0.5,1.3) and (1.5,3.2). Choose between the two fixed candidates using mean squared error on those tuning rows. Keep a separate final-test row at u = 3, target y = 10, outside that choice.

Define residual as target minus prediction, sum of squared errors SSE = Σ residual², and mean squared error MSE = SSE/n for the relevant subset. These are unweighted deterministic response errors. No measurement variance or sampling model is assigned, so they are not automatically chi-squared statistics or estimates with confidence levels.

The order is a teaching protocol, not a claim that a real blind astronomical experiment was performed. Its advantage is that every target, prediction and information role is visible, including the test result that challenges the tuning-based choice.

The better tuning result can predict the new regime worse

Candidate A’s tuning residuals are +0.05 and −0.05. Its SSE is 0.005 and MSE is 0.0025. Candidate B’s tuning residuals are −0.04375 and +0.04375. Its SSE is 0.003828125 and MSE is 0.0019140625.

Under the declared rule, candidate B is selected because it has the lower tuning MSE. The training errors could not have distinguished them, and the tuning rows favour B inside the interval [0,2].

At the separate final-test input u = 3, A predicts 10 and has zero residual. B predicts 11.5, giving residual −1.5 and squared error 2.25. The tuning-selected candidate performs worse on this newly revealed outside-range target.

This does not prove A universally correct or B physically impossible. It demonstrates a specific failure of transferring the tuning ranking to a different input regime. One point cannot estimate a distribution of future errors, and no observational uncertainty is supplied with which to claim statistical significance.

Separate rows can still share the information a test should withhold

Suppose multiple images of each star are randomly divided between training and test. The model may encounter the same source, background pattern or observing artefact in both. That evaluation can be easier than classifying an entirely unseen star, which is a different intended use.

For a new-source question, group the relevant repeated source data appropriately. For a future-time question, preserve time order. For transfer across sites or instruments, examine which calibration, response or preprocessing information is shared. The suitable split follows the scientific target rather than an automatic row-level shuffle.

Fit preprocessing transformations and feature selection within the development procedure. A transformation learned using final-test values or labels can leak information even if the final model coefficients were fitted only to the training rows. Preserve these dependencies in the analysis record.

Different instruments or teams can add useful independent evidence but can still share source catalogues, reference standards or physical assumptions. Multiple names do not establish complete independence. Report which possible errors or explanations the additional measurement constrains.

Retain failure and seek fresh evidence for a revised claim

If the final u = 3 result is used to select A or redesign B, retain it as development history. A fresh independent evaluation of the revised procedure requires information not used in that revision. The original test result remains useful evidence of what the previous selection failed to predict.

Baron’s astronomy review distinguishes training, tuning and final testing. Luo and colleagues provide a concrete dated gamma-ray example in which labelled-source evaluation and later cross-matched sources represent different samples. The latter is an additional comparison, not an automatic unbiased test of every unassociated source.

For a physical model, state the predicted observable, acceptable discrepancy, referent uncertainty and domain before assessing agreement. Residual checks and source or instrument comparisons can reveal different limitations. Agreement over an interpolation range should not be described as universal success in extrapolation.

An informative report preserves the complete ledger, model versions, access history, selection criteria, loss definition and remaining dependencies. The next lesson uses this discipline to examine learned classifiers: how a change in the evaluated population can alter performance even when a specific conditional relation is assumed unchanged.

Worked examples

Try the reasoning

1. If test results repeatedly guide model choices, what role does that information now play?

   Show the answer and reasoning

   Development or tuning information.

   Its filename does not preserve an untouched evaluation role.

2. Why may a row-level split of repeated images overstate performance on unseen stars?

   Show the answer and reasoning

   The same source or shared observing information can occur on both sides.

   The split must match the independent unit relevant to the intended use.

3. Is the toy’s MSE automatically a measurement uncertainty?

   Show the answer and reasoning

   No.

   It is a deterministic descriptive error without a supplied sampling or uncertainty model.

4. Does a good test inside the calibration range establish outside-range performance?

   Show the answer and reasoning

   No.

   Interpolation and transfer to a new regime require different supporting evidence.

What to carry forward

• Separate fitting, selection and final evaluation by actual information use.
• Inspect the complete error ledger and the domain each subset covers.
• Preserve dependencies and obtain fresh tests when a result has guided revision.