Validation, test independence and leakage
Research context
research method · established · As of 2026-10-10
Training fits parameters, validation selects models/tuning, and a final held-out test estimates performance of the selected procedure on its sampling population.
Key knowledge
Training fits parameters, validation selects models/tuning, and a final held-out test estimates performance of the selected procedure on its sampling population. Repeatedly selecting on test results turns that set into validation data. Preprocessing fitted before the split, duplicated subjects, future information or family/site dependence can leak information. Under distribution shift, a correct in-population test estimate does not establish deployment accuracy.
Conditions and scope
- Splits follow the independent unit and time order relevant to deployment, not automatically individual rows.
- Report loss, uncertainty, subgroup population and all tuning access; the untouched test must represent the intended target.
Sources · 1
References
Sources retain their recorded scope. A source link alone does not establish verification of every statement.
Evidence and review
Author checked this English statement, displayed equation where present and stated hypotheses against these source locations: Chapters 2–6 PAC/VC, 11 validation, 13 regularization, 14 stochastic gradient and 23 PCA. The check is bounded to the recorded claims; no complete proof reconstruction, software proof replay or universal literature completeness is claimed.
2026-10-10 · Author source-body/statement/equation/domain check, with graph associations scoped individually. Separate independent content/source/inventory review is required before branch closure.
Entry details
Permanent link to this entry ↗- Record identity
- research-info-validation-leakage
- Revision
- research.1
- Structured reference record
- Postgraduate & research reference