Astronomy · Testing models against the sky

Machine learning, training bias and interpretation

Define what a learned classifier predicts and calculate how its performance changes under a precisely stated shift in the evaluated feature population.

Continue in the approved chapter order. Necessary vocabulary, quantities and assumptions are explained in the lesson.

Back to the chapter

After reading, you can

Identify features, labels, development choices and the intended evaluation population.
Distinguish general dataset shift from a stipulated pure covariate shift.
Calculate separate classification metrics and explain why prediction does not uniquely identify a physical cause.

Features and labels have observational histories

A supervised learning model uses examples with measured features and supplied target values. Features can be spectral measurements, light-curve summaries or image information. A target can be a class label or a continuous quantity. Parameters describe the fitted mapping, while tuning choices help determine how that mapping is learned.

In astronomy, a training label is not automatically an exact statement of nature. It may come from a noisy spectrum, an uncertain counterpart association, a catalogue convention or a simulation that omits some physics. Its provenance and uncertainty matter for interpreting what the learned model predicts.

The labelled sample also depends on which sources were detected and successfully characterized. Bright, isolated or otherwise easy-to-identify sources can be represented differently from the objects awaiting classification. Noise, missing classes, observing configuration and preprocessing can all affect the mapping and its measured performance.

Fit preprocessing and feature selection inside the development procedure, preserving the independent units and final-test roles from the preceding lesson. A random split from one selected catalogue can test performance in that population without establishing performance for another population.

Pure covariate shift is a particular assumption

Let X denote measured features and Y a target label. General dataset shift means the joint law p(X,Y) changes between source and target populations. Pure covariate shift is the more specific condition that p(X) changes while p(Y | X) remains the same.

Different feature histograms alone do not establish an unchanged conditional label law. New source kinds, changed instruments, unreliable labels or altered observing conditions can change more than the feature marginal. Such changes require additional modelling and tests rather than automatic use of a covariate-shift correction.

For an original exact toy, allow only two feature states, H and L. They are abstract categories, not measured bright and faint survey groups. Fix a classifier that predicts class 1 for H and class 0 for L. In both populations stipulate P(Y = 1 | H) = 0.98 and P(Y = 0 | L) = 0.60.

Let the source-like population have P(H) = 0.9 and P(L) = 0.1, while the target population has P(H) = 0.2 and P(L) = 0.8. The unchanged conditional law explicitly makes this a pure covariate-shift example. The classifier itself is fixed; no model is trained or deployed here.

The same classifier can make a different mixture of errors

Use expected counts per 1000 cases. A true positive is predicted class 1 and actual class 1; a false positive is predicted class 1 and actual class 0. A true negative is predicted and actual class 0, while a false negative is predicted class 0 and actual class 1.

In the source-like population, the expected 900 H cases give 900 × 0.98 = 882 true positives and 18 false positives. The expected 100 L cases give 100 × 0.60 = 60 true negatives and 40 false negatives.

In the target population, the expected 200 H cases give 196 true positives and four false positives. The expected 800 L cases give 480 true negatives and 320 false negatives. Both rows sum to 1000; they are stipulated expectation tables rather than observed confusion matrices.

The conditional success rates within H and L have not changed. Their population weights have. The target contains many more cases in the feature state for which this classifier predicts class 0 even though the actual class is 1 with probability 0.4.

Original expected confusion counts per 1000 cases under the fixed classifier and explicitly unchanged conditional class law.
Evaluation population	True positive	False positive	True negative	False negative
Source-like: P(H) = 0.9	882	18	60	40
Target: P(H) = 0.2	196	4	480	320

Accuracy, precision and recall are distinct

Accuracy is the fraction classified correctly: (TP + TN)/total. It is 942/1000 = 0.942 for the source-like population and 676/1000 = 0.676 for the target population. A single accuracy reported without its evaluation population hides this change.

Precision for predicted class 1 is TP/(TP + FP). It remains 882/900 = 0.98 and 196/200 = 0.98 because the classifier predicts class 1 exactly in state H and its conditional label law was fixed. This invariance is a property of the supplied toy, not a universal consequence of covariate shift.

Recall for actual class 1 is TP/(TP + FN). It changes from 882/922 ≈ 0.956616 to 196/516 ≈ 0.379845. Thus unchanged precision can coexist with a much larger fraction of actual class-1 cases missed in the target population.

Choose metrics in relation to the scientific aim and the consequences of different errors. A selected candidate list, a population census and a limited follow-up budget may need different tradeoffs. Report the class definitions, population, thresholds, subgroup behaviour and uncertainty of real performance estimates.

Metrics derived from the two expected confusion rows. They are conditional toy results, not performance claims for a real astronomical catalogue.
Population	Accuracy	Class-1 precision	Class-1 recall
Source-like	0.942	0.98	0.956616
Target	0.676	0.98	0.379845

Reweighting needs support and a justified conditional law

If the conditional law is unchanged and target feature states are represented in the source population, weighting can help evaluate a target-like distribution. A density ratio cannot recover arbitrary information about a target region with no training support. It also does not automatically repair changed conditional laws or incorrect labels.

Luo and colleagues’ gamma-ray study compares evaluation drawn from identified sources with a later cross-matched sample. Malyshev’s study explicitly distinguishes pure covariate shift and discusses the limits of that comparison: cross-matched sources can remain selective, and reconstructed measurement features can change. These are dated methodological examples, not present catalogue classifications or current algorithm rankings.

A useful transfer test retains the original source and target definitions, measurement and label routes, splitting decisions and intended use. Evaluate whether the test actually samples the difficult sources to which predictions will be applied. An uncertainty score or an out-of-distribution flag also needs testing under that use.

Different instruments, noisy features and repeated measurements can create dependencies not captured by a simple catalogue split. More labelled examples from the same easy subset do not automatically provide evidence about missing classes or absent target regimes.

Interpret the mapping at the level the evidence supports

A class score describes a learned mapping conditional on its training information and model. Feature importance or saliency identifies aspects influential in that mapping; neither uniquely determines the causal emission mechanism. A correlated observational feature can be useful for prediction without being the physical cause of the class.

Malyshev’s gamma-ray analysis gives a concrete distinction between groups constructed from similar measured features and physical source categories. Physically different sources can be combined in one predictive group when the available gamma-ray properties do not separate them. An accurate group prediction does not recover information absent from those properties.

Independent counterpart evidence, timing signatures, spectra, spatial structure and physical source modelling can strengthen an interpretation, subject to their own calibration and selection conditions. State what the classifier contributes to that wider argument rather than treating it as a replacement for every observation.

Learned prediction is useful when its target, evaluation population and interpretation are explicit. The next lesson returns to a general modelling task: using a structured failure to investigate assumptions, revise a bounded prediction and obtain new evidence for the revision.

Worked examples

Compute target-population errors

What are the expected confusion counts per 1000 target cases with P(H) = 0.2 and the stipulated fixed conditional law?

The 200 H cases give 196 true positives and four false positives.
The 800 L cases give 480 true negatives and 320 false negatives.
Check that 196 + 4 + 480 + 320 = 1000.

TP = 196, FP = 4, TN = 480 and FN = 320 in expectation.

Compare two class-1 metrics

Why is target precision 0.98 while target recall is only about 0.379845?

Precision uses predicted positives: 196/(196 + 4) = 0.98.
Recall uses actual positives: 196/(196 + 320) ≈ 0.379845.
Many actual positives occur in L and are predicted class 0.

The denominators answer different questions. High precision of the retained positive predictions does not imply recovery of most actual positives.

Try the reasoning

What is held unchanged by the stipulated pure covariate shift?
Show the answer and reasoning
The conditional label law p(Y | X).
The feature marginal changes, while the conditional law is explicitly assumed fixed.
Do different feature histograms alone prove pure covariate shift?
Show the answer and reasoning
No.
They do not establish that the conditional label law is unchanged.
Can density weighting alone recover arbitrary target information where the training population has zero support?
Show the answer and reasoning
No.
The required source information is absent in that domain.
Does high feature importance uniquely establish an astronomical causal mechanism?
Show the answer and reasoning
No.
It describes the learned predictive mapping and needs separate physical evidence for causal interpretation.

What to carry forward

Training features and labels inherit observing, selection and calibration conditions.
Specify the evaluation population and report distinct metrics under transfer.
Prediction, reweighting and feature importance do not automatically identify a unique physical source mechanism.

References

Teaching resources and research supporting this lesson’s explanations and examples.