Astronomy · Evidence for planets around other stars

False positives, confirmation and detection biases

Separate selecting a candidate from establishing its interpretation, then use conditional recovery and explicit denominators to understand why detected fractions differ from underlying planet frequencies.

Continue in the approved chapter order. Necessary vocabulary, quantities and assumptions are explained in the lesson.

Back to the chapter

After reading, you can

Distinguish candidate selection, validation and parameter measurement.
Explain why recovery of injected signals does not measure every kind of reliability.
Calculate selection and classification fractions with the appropriate conditioning and denominators.

Name the evidence stage and the alternatives being tested

A candidate is a signal selected for further assessment under a search procedure. A validation analysis compares a planetary interpretation with specified alternatives using data and assumptions. Additional dynamical or other evidence can strengthen the interpretation and constrain physical parameters. These stages should not be collapsed into one label.

Possible alternatives include eclipsing stars, blended sources, stellar activity, detector effects and reduction artefacts. Which tests matter depends on the signal and observing setup. A model comparison that excludes instrumental errors does not supply a probability covering those errors.

Terminology and decision thresholds vary across investigations. State what a reported probability or confirmation claim actually includes: the data, model alternatives, population assumptions and version of the analysis. Statistical confidence in one defined comparison is not an inventory of every possible failure.

A later radius or atmosphere estimate is another inference stage. Even a well-supported planetary companion can have uncertain properties. Confirming the physical cause of a signal does not make all subsequent parameter estimates equally secure.

Combine light before deciding what the depth means

Consider an original additive-flux counterexample. In one band, a constant source contributes 98 units and an unresolved eclipsing system contributes 2 units out of eclipse. If that system loses half its light, the combined flux falls from 100 to 99 units.

The observed depth is one percent, the same as the ideal k = 0.10 planetary transit in the earlier lesson. This bookkeeping does not describe a complete stellar binary, but it demonstrates that identical depth alone cannot choose between physical hypotheses.

Event shape, alternating eclipse patterns, spectral information and image-position changes can help test such alternatives. A repeated signal should be checked against sampling gaps and other periods. There is no universal rule that a particular fixed number of dips excludes all nonplanetary causes.

Complementary observations add information to the extent that they constrain different possibilities. Products sharing the same exposures, calibration or stellar model can retain shared errors. Count the independent constraints supplied, rather than simply counting files or analysis pipelines.

Selection acts on a population before a candidate list appears

A transit survey selects systems with suitable orientation as well as detectable sampled signals. For an isotropic distribution of circular orbital orientations, geometric transit probability including grazing cases is p_geo = (R_star + R_p)/a, provided the bodies are separated and this ratio is at most one. All lengths must use the same units.

Let recovery mean selection by a stated observing and search process, conditional on a geometric transit and a defined signal population. Then total selection probability for an existing planet is p_geo times that conditional recovery probability. This is conditional probability bookkeeping, not an assumption that every noise, cadence and window effect is independent.

Use two synthetic groups, each with 1,000 targets and a stipulated 500 true planet hosts. Set R_star = 1 and R_p = 0.1 in common arbitrary length units. Group A has circular a = 10 and conditional recovery 0.8; group B has a = 20 and conditional recovery 0.4.

The resulting selection probabilities are 0.11 × 0.8 = 0.088 and 0.055 × 0.4 = 0.022. Expected genuine selections are therefore 44 and 11. The same underlying host fraction of one half produces different expected selected fractions, 0.044 and 0.011.

Original conditional expectation model with no false candidates. Both groups have 500 true hosts among 1,000 targets; these are not observed survey counts.
Group	p_geo	Recovery given geometric transit	Expected genuine selections
A	0.11	0.8	44
B	0.055	0.4	11

Test sensitivity within the population actually simulated

Injection–recovery tests add declared synthetic signals to a specified data or processing stage and measure which the pipeline recovers. The result depends on target properties, signal parameters, noise, sampling and implementation. It is a sensitivity estimate for that tested distribution.

Recovery does not by itself measure how many selected real candidates are false positives. A pipeline may recover many injected transits while also selecting instrumental artefacts or eclipsing stars. Validation of the selected population and recovery of true test signals answer different questions.

In the two-group expectation example, dividing each expected selected fraction by its correct total selection probability returns the stipulated host fraction 0.5. This algebra is not a universal estimator or an uncertainty calculation. Real inference must handle false candidates, uncertain recovery, correlations, target selection and regions with little or no sensitivity.

An occurrence-rate claim therefore needs a population model and a documented selection function. A catalogue’s raw detection fraction is a property of both the population and the search. It cannot be read directly as the fraction of all stars with those planets.

The denominator changes the question

For a separate original classification example, stipulate full knowledge of 1,000 target labels. There are 100 true planet targets and 900 nonplanet targets. The procedure selects 80 of the planets and 18 of the nonplanets, leaving 20 missed planets and 882 rejected nonplanets.

Recovery among true planets is 80/100 = 0.80. The false-positive rate among nonplanets is 18/900 = 0.02. Candidate reliability is instead 80/(80 + 18) ≈ 0.8163, while the false fraction of the candidate list is 18/98 ≈ 0.1837.

Two percent is thus not the false fraction among candidates in this model. The target base rates and both selection behaviours determine the selected mixture. A probability conditioned on a nonplanet target differs from one conditioned on membership in the candidate list.

The table is classification arithmetic with supplied truth labels, not a validation probability for any real candidate. In practice, truth is incompletely known and reliability must be estimated under specified alternatives. The next lesson assumes suitable evidence and asks how different observables constrain size, mass and orbit.

Original fully labelled classification table. Rows describe true target type; columns describe selection.
True type	Selected	Not selected	Total
Planet target	80	20	100
Nonplanet target	18	882	900
Total	98	902	1000

Worked examples

Apply the conditional selection probability

Why does group A have more expected genuine selections despite the same true host fraction?

Its geometric probability is 1.1/10 = 0.11 and conditional recovery is 0.8.
Total selection probability is 0.088, giving 500 × 0.088 = 44 expected selections.
Group B instead gives 500 × 0.022 = 11 because its geometry and conditional recovery differ.

Selection differs. The raw expected detection fractions do not establish different true host fractions.

Condition on the selected list

What fraction of the 98 selected candidates in the separate classification table are true planets?

The selected list contains 80 true planets and 18 nonplanets.
Reliability within this stipulated list is 80/98.
This is approximately 0.8163, not 1 − 0.02.

About 81.63 percent in the invented fully labelled example. Its denominator is the selected list.

Try the reasoning

Does a one-percent dip alone establish a planetary interpretation?
Show the answer and reasoning
No.
The additive eclipsing-blend example has the same depth; additional evidence must test alternatives.
Does recovery of injected true signals by itself give candidate reliability?
Show the answer and reasoning
No.
Recovery conditions on true simulated signals; reliability conditions on the selected candidate list.
Is the classification example’s 2 percent nonplanet false-positive rate equal to its false-candidate fraction?
Show the answer and reasoning
No.
The latter is 18/98 ≈ 18.37 percent; the denominators differ.
Does a reported astrophysical false-positive probability automatically include all instrumental failures?
Show the answer and reasoning
No.
Its meaning is restricted to the stated hypothesis space, inputs and modelling assumptions.

What to carry forward

A selected candidate and a tested planetary interpretation are different evidence stages.
Selection probabilities are conditional on geometry, targets, signals and the observing pipeline.
Completeness, false-positive rate and candidate reliability require different denominators and declared assumptions.

References

Teaching resources and research supporting this lesson’s explanations and examples.