Astronomy · Planning observations around a question

Predicted signatures, upper limits and stopping criteria

Construct a preplanned Poisson detection rule, distinguish its recovery capability from an observed-source bound, and state when a changed stopping rule needs new calibration.

Continue in the approved chapter order. Necessary vocabulary, quantities and assumptions are explained in the lesson.

Back to the chapter

After reading, you can

Specify a detection statistic, fixed observing duration and false-positive criterion.
Calculate detection power and a procedure-defined limiting flux.
Distinguish a nondetection, an inferred upper bound and the consequences of repeated testing or adaptive stopping.

A search needs a rule before its result is interpreted

A predicted signature connects a physical proposal to a recorded quantity in a specified band, direction and time window. To interpret a search, also state its background model, detection statistic and decision rule. Without those definitions, “nothing significant appeared” does not say which sources the search could have detected.

A false positive is a declared detection when the null model is true; a false negative is a missed detection under a specified source alternative. Detection power is the probability of declaring detection under that alternative. These are sampling properties of a procedure, not posterior probabilities that a particular source exists.

The term upper limit has several conventions in astronomy. Kashyap and colleagues distinguish a procedure’s ability to detect a source at specified power from an upper bound inferred for the intensity of a particular observed source. Read and state the convention rather than treating the words as interchangeable.

Here we will construct an original detection-capability example. It supplies no observed source count and produces no confidence or credible bound for a real object. The calculation instead answers what steady flux would give a chosen recovery probability in a preplanned search.

Choose the smallest count threshold meeting the false-positive bound

Stipulate a known stationary Poisson background rate b = 0.002 counts s⁻¹ and a fixed 100-second exposure. Under the no-source null model, N has mean 0.2. Set a maximum false-positive probability of 0.005 for this one preplanned test.

Choose the smallest integer k for which P(N ≥ k | source absent) ≤ 0.005, and declare detection only when N ≥ k. A Poisson upper tail is 1 − e^(−λ)Σj=0 to k−1 λʲ/j!, with the indicated count convention.

At k = 2 the null tail is approximately 0.017523096, which exceeds the bound. At k = 3 it is approximately 0.001148481, which satisfies it. The rule is therefore N ≥ 3 for this fixed design.

The achieved null probability is below 0.005 because counts are discrete. It need not equal the requested bound exactly. Writing N > 3 instead would be a different integer threshold. Preserve whether the detection condition is strict or inclusive when reproducing a procedure.

A detection threshold and a limiting flux are different quantities

Let a constant band-integrated photon flux F pass through a fixed effective area A = 10 cm², including all stipulated acceptance losses. The source count mean in duration t is μs = FAt. Assume its counts and the background are independent Poisson arrivals and that the response and background are known.

For the 100-second design, the detection power is P(N ≥ 3 | λ = 0.2 + μs). Set the desired power to 0.9. Solving the Poisson tail gives total mean approximately 5.322320338 and source mean approximately 5.122320338.

The corresponding flux is F90 = 5.122320338/(10 × 100) ≈ 0.00512232034 photons cm⁻² s⁻¹. It is the flux giving 90% detection probability under this procedure. At that flux, 10% of realizations still fail to meet the threshold.

For illustration, flux 0.001 photons cm⁻² s⁻¹ gives one expected source count and power about 0.120512901. Flux 0.005 gives five expected source counts and power about 0.891213350. A real source can therefore be missed; the word nondetection cannot turn the procedure’s recovery probability into certainty about its absence.

A longer exposure changes both the background and the decision threshold

Now consider a separately preplanned 200-second design under the same rates and area. The background mean becomes 0.4. At k = 3 the null tail is approximately 0.007926332, which is too large; at k = 4 it is approximately 0.000776251, which meets the same 0.005 bound.

With the rule N ≥ 4, 90% power requires total mean approximately 6.680783068, hence source mean approximately 6.280783068. The needed source mean is larger than for the shorter test, but it is collected over twice the duration. The limiting flux is approximately 0.00314039153 photons cm⁻² s⁻¹.

Thus the longer design reaches a lower steady flux at the stated recovery probability. It does not follow by retaining the old count threshold or simply halving the old flux: the background and discrete decision rule have changed too.

These are two alternative fixed designs, each calibrated separately. The table does not calibrate a procedure that first inspects the 100-second result and then decides whether to continue. That procedure uses data differently and needs its own statistical analysis.

Original separately preplanned fixed-duration searches. Known background rate is 0.002 counts s⁻¹, effective area 10 cm², false-positive bound 0.005 per test and desired power 0.9.
Duration (s)	Background mean	Detection rule	Actual null false-positive probability	Source mean for 90% power	F90 (photons cm⁻² s⁻¹)
100	0.2	N ≥ 3	0.001148481	5.122320338	0.00512232034
200	0.4	N ≥ 4	0.000776251	6.280783068	0.00314039153

Interpret a nondetection using the quantity actually calculated

F90 is a property of the stipulated detection process and source model. It was computed before seeing a source count. An upper confidence bound or posterior credible bound for a particular object instead uses its recorded data and the corresponding statistical construction.

If an actual count is below the threshold, report the nondetection with its searched band, time, coverage and response. If a source bound is desired, construct it from the observation and assumptions, including background or calibration uncertainty where relevant. Do not label F90 a 90% posterior bound merely because its power is 90%.

The effective area is fixed in this toy. Real spectra, energy-dependent response, observing gaps, spatial searches, unknown background, dead time and correlations can change both false-positive and recovery probabilities. The metrological definition of detection limit likewise depends on the procedure and its false-presence and false-absence probabilities.

Even notation needs care: some sources use β for detection power and others for the miss probability. Writing “power 0.9, miss probability 0.1” avoids that ambiguity. A system’s sensitivity in the metrological response-slope sense is also different from this probability-defined detection capability.

Repeated looks are part of the procedure

Set the intended exposure, endpoints and maximum effort before interpreting results, or specify a valid adaptive design from the outset. Record which circumstances would trigger an extension, a new target, a changed band or a separate confirmation observation.

A per-test false-positive bound does not automatically control the chance of any false positive across many targets, sky positions or repeated looks. The data can be correlated and the trials need not be independent. The relevant procedure must account for the searches and decisions actually performed.

Howard and colleagues describe confidence sequences with coverage guarantees uniform over time under their stated hypotheses. Such methods illustrate that sequential guarantees are a distinct statistical property, not a label acquired by repeatedly applying an ordinary fixed-time interval or threshold. No sequential procedure is constructed in this lesson.

A stopping decision can also reflect scientific or resource limits without being a detection claim. Preserve that reason alongside the data and inference. The next chapter turns to validation: whether an implemented model and its tests support the scientific use being proposed.

Worked examples

Check the fixed 100-second threshold

Why is k = 3 the chosen inclusive threshold under the stated 0.005 false-positive bound?

The no-source count mean is 0.002 × 100 = 0.2.
P(N ≥ 2 | 0.2) ≈ 0.017523096, above the bound.
P(N ≥ 3 | 0.2) ≈ 0.001148481, below it.

Three is the smallest qualifying integer, so this design declares detection at N ≥ 3.

Convert the 200-second source mean into flux

What flux gives 90% power for the separately preplanned 200-second rule N ≥ 4?

The required total count mean is approximately 6.680783068.
Subtract the known background mean 0.4, obtaining source mean 6.280783068.
Divide by area × duration = 10 cm² × 200 s.

Approximately 0.00314039153 photons cm⁻² s⁻¹, a procedure-defined limiting flux rather than an observed-source upper bound.

Try the reasoning

Can a source at the supplied F90 fail to be detected?
Show the answer and reasoning
Yes, with probability 0.1 under the stated model.
Power 0.9 is a recovery probability, not a guarantee.
Must the actual false-positive probability equal the chosen 0.005 bound?
Show the answer and reasoning
No.
Discrete count thresholds can achieve a smaller probability.
Does a power-defined F90 automatically give a 90% posterior upper bound after a nondetection?
Show the answer and reasoning
No.
A source bound requires the observed data and its own statistical construction.
Does the fixed 100-second rule retain its overall error guarantee when repeatedly reused at data-dependent stopping times?
Show the answer and reasoning
No automatic guarantee follows.
The sequential procedure and its hypotheses need separate calibration.

What to carry forward

Define the searched signature, duration, statistic and decision rule.
Separate the count threshold, detection power and an upper bound for an observed source.
Stopping, extensions and multiple searches belong in the statistical procedure.

References

Teaching resources and research supporting this lesson’s explanations and examples.