Reproducible workflows, prediction, validation and machine-learning limits

Define the claim before interpreting a prediction

A computational workflow applies stated transformations to defined inputs. Recomputing its output checks the computational account. Predictive validation instead asks whether a chosen rule works on evidence not used to choose that rule, for a specified target. Neither result automatically establishes the mechanism that generated the pattern.

A target might be a new reading from a previously observed unit, a new individual from the same population, or a new population under different conditions. These are different tasks. The original split model asks about new independent units. Sharing a unit between training and testing therefore does not supply the intended held-out evidence.

A reproducible workflow retains input identities, reference and annotation versions, transformations, settings and the chosen evaluation account. That record helps others identify what was done. It cannot by itself repair an inappropriate split, biased test population or a target that is only an imperfect proxy for the biological question.

Information used for choosing a rule belongs on the training side

Training uses data to choose a predictive rule. Validation can guide choices such as which candidate model to retain. A held-out test assesses the chosen procedure under its stated target. If test information influences feature selection or other learned transformations, the eventual test is no longer independent of those choices.

Leakage can be subtle. Separate files can contain the same or dependent biological material; a predictor can include information unavailable at the intended prediction time; or features can be selected using outcomes from the whole dataset before a split. Repeating a contaminated assessment faithfully does not make its performance estimate appropriate.

In the model's first split, one reading from each of eight units trains and the other tests. All eight unit identities are shared. In the second split, all readings from units one through four train and all readings from units five through eight test. It removes this particular unit-overlap problem under the stipulated independence, without guaranteeing that every other design issue is resolved.

A passing test has a population and a metric

A classification account compares supplied predictions with known categories. Accuracy divides correct predictions by all evaluated cases. Sensitivity divides correctly identified positives by all cases in the positive category; specificity divides correctly rejected negatives by all cases in the negative category. These denominators answer different questions. A single percentage can hide which cases were missed.

The separate eight-unit test has three correctly identified positives, one missed positive, three correctly identified negatives and one false positive. Accuracy is six/eight. A second supplied environment record has different prediction errors and accuracy four/eight. These are original assigned outcomes, not measured results of a trained algorithm or evidence that an environmental change caused the difference.

Independent testing in one population does not guarantee transport to another. Recruitment, measurement, conditions or category definitions may differ. Testing in the intended setting addresses this uncertainty, while preserving how that setting was selected. Successful prediction still leaves causal explanations open: a useful proxy can forecast an outcome without being its mechanism.

Worked example

Check your understanding

1. Why is a file-wise split insufficient for the model's new-unit target?

   Show reasoning

   Both sides contain readings from all eight biological units. The test concerns another reading of already represented units, while the stated claim concerns new units. File-level separation does not remove the dependence.

2. Why must outcome-informed feature selection be included within the training part of validation?

   Show reasoning

   Selecting features using test outcomes gives the chosen rule information from its eventual assessment. Holding out the final fitting step alone does not undo that earlier use of test information.

3. Does accuracy three quarters on the primary toy test establish the same accuracy in another population or a causal mechanism?

   Show reasoning

   No. Performance belongs to the stated records, metric and population. The second supplied record illustrates a different result; neither record supplies a generalisation guarantee, intervention or mechanistic proof.

What to carry forward

A reproducible computation and a valid prediction claim require different checks; define the units, split, metric and population before generalising.