Model verification, validation and uncertainty

Check the implementation and the representation

Model verification examines whether the implemented computation follows its specification and solves the stated mathematical problem adequately. Independent hand calculations, known special cases and numerical refinement can reveal mistakes. Repeating the same mistaken formula in a second spreadsheet is a weak independent check. Look for a reference or reasoning path that can fail differently.

Model validation examines correspondence with the system represented for an intended use and domain. Compare suitable observations or an acceptable representative system with the model response. A model can be verified but unsuitable for a physical purpose. A close match in one case can also conceal compensating errors or provide little evidence about a different condition.

Tuning and testing serve different roles

Suppose a parameter is chosen to fit one set of observations. That adjustment is model fitting, sometimes called model calibration; it is distinct from calibrating a measuring instrument. The fitted observations are evidence about the fitting process, but agreement with them is weaker evidence about prediction of new cases. Preserve separate, relevant cases when assessing predictive performance.

Our constructed held-out temperature-rise predictions are 8.0, 10.0 and 12.0 K; observations are 8.1, 9.8 and 11.2 K. Prediction minus observation gives−0.1,+0.2,+0.8 K. With a preselected classroom discrepancy criterion of absolute residual no greater than 0.5 K, the third case fails. Reporting only the first two would hide the limitation.

A validation conclusion needs conditions

The criterion in this example is deliberately simple. A real comparison must consider uncertainty in observations, uncertain inputs, numerical error, model discrepancy and relevance to the decision. An uncertainty interval is not automatically a guaranteed physical range. Nor does a small residual prove that every uncertainty contribution is small: different errors can cancel.

State what was tested, what metric was used, the conditions covered and what remains untested. Evidence about slow changes at moderate temperatures does not automatically cover rapid transients or a different material. A changed input range or intended decision may require further work. Confidence should be attached to the specified use, with its evidence, rather than to a software name or an unqualified “validated” label.

Read the failed case rather than averaging it away

For case C, 12.0−11.2=+0.8 K, so the prediction is 0.8 K higher than the observation. Its absolute residual exceeds 0.5 K by 0.3 K. The signed mean of all three residuals is 0.3 K, but that mean does not make C satisfy the per-case criterion.

Investigate the discrepancy, uncertainty and relevant conditions before selecting a response. Changing the criterion after seeing C would change the question being judged. If the model is revised using C, label that use and seek appropriate further evidence instead of continuing to call C an untouched predictive test.

Try the reasoning

1. Does numerical convergence establish physical validation?

   Show the answer and reasoning

   No. It addresses a numerical solution of an assumed mathematical model. Correspondence with the represented system requires separate evidence.

2. Can a mean residual hide a troublesome case?

   Show the answer and reasoning

   Yes. Aggregation can conceal large individual errors and cancellation. Choose and report metrics appropriate to the intended decision.

3. Why record the validation domain?

   Show the answer and reasoning

   It identifies the conditions supported by the evidence and helps recognise a proposed use that extends beyond them.

What to carry forward

Keep the implemented mathematics, observed system and intended decision linked by distinct, documented checks.