Chemistry · Lesson

Structure-property models, machine learning and applicability limits

Evaluate invented predictions using explicit target definitions, training boundaries, originating groups and finite coverage.

Back to the Chemistry map

By the end, you can

Calculate bias, MAE and RMSE with a stated residual sign.
Fit an inductive preprocessing transformation using training information.
Separate shared-origin groups for the intended prediction task.
Distinguish finite observed coverage from independently validated calibration.

A structure-property model needs a defined prediction target

A model relates selected molecular or material features to a specified property in a defined data set. The target’s units, measurement context and chemical identity are part of the task. Features and labels require consistent preparation and provenance.

A structural descriptor can omit information relevant to the property. A convenient representation or a strong association does not automatically establish a causal chemical explanation.

Evaluation starts with a defined target and a fixed transformation

A chemical prediction target needs its quantity, entity state, units and conditions. A numerical label called energy may mean electronic energy, a particular reaction energy or free energy. The invented transfer-free-energy cases below specify the same solvents, temperature and convention before comparing residuals.

Signed bias measures average direction; MAE and RMSE measure different aspects of magnitude. A small bias can conceal large opposite-sign errors. The four-entry arithmetic describes those entries rather than certifying performance on a broader chemical population.

For the prescribed inductive feature example, fit the mean and scale from training values and then apply the unchanged transformation to the query. Letting the query alter the fitted scale changes what was held out. Explicit transductive designs can use different rules, but they need to declare the intended evaluation question.

Opposite signs partly cancel in bias

Symbolic entity	Reference / kJ mol⁻¹	Prediction / kJ mol⁻¹	Residual / kJ mol⁻¹
A	−12	−10	+2
B	−4	−7	−3
C	6	10	+4
D	14	13	−1

Original four-entity transfer-free-energy evaluation at the assigned common conditions. Each residual is prediction minus reference. These invented rows contain no actual compound measurements.

Independent evaluation must avoid information leakage

Training data determine fitted parameters and learned preprocessing. Evaluation data test the resulting prediction under a defined separation from training. Related structures, replicated measurements and shared experimental units can undermine independence if distributed carelessly between sets.

Fit transformations using the training information appropriate to the task. A held-out label used to choose settings is no longer a wholly independent final test. The division should match the intended future prediction, rather than merely produce a random-looking split.

The held-out unit should match the intended use

Rows derived from one originating unit can share information. A row-level split may put related information on both sides while appearing to reserve a test set. Holding out complete origin groups answers a more demanding new-origin prediction question. Additional chemical, structural or temporal dependencies require their own checks.

Prediction intervals also need a stated target and evaluation basis. Count how many held-out reference values fall within the defined intervals, retaining endpoints and residual sign conventions. Eight entries yield a descriptive fraction; they cannot guarantee the advertised long-run coverage.

If a width is revised using the test residuals, those entries have informed the decision. Freeze the revision and seek suitable new independent evidence for its performance. A single heavy-atom descriptor, convenient score or nominal percentage does not establish a universal applicability domain.

A prediction error belongs to its evaluation population

Bias and spread of residuals describe different aspects of error. Define the residual sign and averaging convention before computing a metric. Several metrics can emphasize different mistakes or scales.

A good average result can hide large errors for a subgroup or a new chemical region. An applicability domain states the conditions under which performance evidence supports use; it is not guaranteed by the model returning a number.

A model’s uncertainty needs calibration and context

An uncertainty estimate can describe selected variability or model limitations under its method. Test whether it is informative on relevant independent cases. A narrow interval can still be misleading when the system is outside the represented range or the data are biased.

The original modelling examples below demonstrate split, error and boundary reasoning. They do not validate a new chemical prediction service or infer a medical, environmental or purchasing decision from a synthetic data set.

Worked examples

Signed bias can hide opposing errors

For four distinct symbolic entities, assign reference transfer free energies [−12,−4,6,14] and predictions [−10,−7,10,13] kJ mol⁻¹ between the same stipulated solvents at 300.0 K and a common standard convention. This is an invented independent evaluation set. Use prediction minus reference for the residual sign.

Residuals are [2,−3,4,−1] kJ mol⁻¹. Their signed mean, the bias, is (2−3+4−1)/4=0.500 kJ mol⁻¹.
Mean absolute error is (2+3+4+1)/4=2.50 kJ mol⁻¹. Root mean squared error is √[(4+9+16+1)/4]=2.73861279 kJ mol⁻¹.
The signed mean is smaller because opposite signs partly cancel. Neither metric establishes performance outside these four assigned cases, and no real solvation values or trained model are supplied.

Bias 0.500, MAE 2.50 and RMSE 2.73861279 kJ mol⁻¹ answer different questions.

Fit a transformation from the training information

For a prescribed inductive evaluation, use heavy-atom counts [4,6,8,10] from four hypothetical training entities. Define preprocessing mean and standard deviation using the population divisor n. Evaluate a new count 18. What changes if that evaluation input is included when fitting the scale?

Training mean is seven; training variance is [(4−7)²+(6−7)²+(8−7)²+(10−7)²]/4=5. The prescribed scale is √5=2.23606798, not an unbiased sample-variance estimate.
Proper query z=(18−7)/√5=4.91934955. If all five values fit the transformation, mean becomes 9.20, scale 4.83321839 and query z=1.82073295.
The smaller apparent distance results from allowing evaluation information to change the transformation. Heavy-atom count alone does not define chemical identity or a validated applicability domain. No target labels or fitted predictor are present.

The training-only query z is 4.91934955; fitting with the query changes it to 1.82073295.

Hold out the origin relevant to the prediction question

Invent groups A:[A1,A2,A3], B:[B1,B2], C:[C1,C2,C3], D:[D1,D2]. Within each letter, rows share one originating experimental unit. A split trains on A1,A2,B1,C1,C2,D1 and tests on A3,B2,C3,D2. Does it test prediction for wholly unseen originating groups?

There are six training rows and four test rows, but each test row shares its originating group with training. Shared groups are A,B,C,D.
For the intended new-group prediction question, an alternative trains on all A/B rows and tests on all C/D rows: five rows each, with no shared origin group.
Origin separation addresses the specified shared-unit leakage. Structural relatives, time, chemical state or other dependencies can need additional split design. Row counts alone do not establish predictive performance.

The first split shares all four origins; the A/B versus C/D split holds out complete origin groups.

Observed coverage is a finite descriptive result

Eight invented independent evaluation residuals are [−0.8,0.4,0.9,−1.1,1.4,−1.9,0.6,2.3] kJ mol⁻¹. Symmetric prediction intervals have half-width 1.50 kJ mol⁻¹ and include their endpoints. They were advertised with nominal fraction 0.900. Find the observed coverage.

An entry is covered when |prediction−reference|≤1.50 kJ mol⁻¹. Six of eight meet this rule; −1.9 and +2.3 do not.
Descriptive coverage is 6/8=0.750. This finite fraction does not guarantee future coverage or by itself provide a sampling interval or a formal rejection test for the advertised fraction.
Choosing a wider interval after inspecting these entries uses their information. That frozen choice needs suitable new independent evaluation for final calibration evidence.

Six of eight entries are covered: observed fraction 0.750.

Try it yourself

Add +2.0 kJ/mol to all four predictions. Recalculate bias, MAE and RMSE.
Show the answer and reasoning
Residuals [4,−1,6,1]; bias 2.50, MAE 3.00 and RMSE 3.67423461 kJ mol⁻¹.
A uniform offset changes the signed residuals and their magnitudes; the three metrics are not interchangeable.
Use test count 14 instead of 18 with the same four training values. Find the proper z and the leaky all-five z.
Show the answer and reasoning
Training-only z=3.13049517; fitting all five gives mean 8.40, scale 3.44093011 and query z=1.62746694.
Fit the mean and scale from training information; evaluate the new count with that fixed transformation.
Hold out only group D and train on A, B, C. Count training/test rows and shared origin groups.
Show the answer and reasoning
Eight training rows, two test rows and no shared originating groups.
Grouping changes both the row counts and the prediction question; the unseen group has no rows in training.
Widen the half-width to 2.00 kJ/mol after inspecting these same eight residuals. Find descriptive coverage and state what further independent test is needed.
Show the answer and reasoning
Seven of eight covered, fraction 0.875; test the frozen wider interval on suitable new independent data.
Seven entries fall within the wider interval, but reusing the inspected set does not independently establish the advertised coverage.

What to carry forward

Define the chemical target before interpreting a numerical score.
Bias, absolute error and squared error summarize different features.
Hold out the information and originating units relevant to the intended prediction task.
An interval chosen using test data needs fresh independent calibration evidence.

References

Teaching resources and reference material supporting this lesson’s explanations and examples.