Multiple tests and unexpected patterns

The procedure is part of the probability statement

A pattern evaluated at one predeclared location answers a different question from the largest pattern found after searching many locations. Its probability statement must describe which procedure produced the reported statistic.

In cosmology, a search can vary sky region, smoothing scale, data selection or statistic. These choices may be strongly correlated, so their number alone does not establish an independent trial count.

A local probability evaluates a specified test at a specified location under a stated null law. A global probability evaluates a declared search statistic under that same law and its complete search procedure.

The finite experiment below supplies every outcome and selection rule. It demonstrates the distinction without assigning a significance or physical explanation to any observed cosmic pattern.

Define the null law before counting exceedances

Under the supplied null, draw 12 independent fair bits. Each of the 4096 ordered twelve-bit outcomes has probability 1/4096. Partition them into four disjoint blocks of three bits each.

Let K_j count heads in block j. The predeclared local exceedance event is K_j≥3, so it occurs only for three heads and has probability p_local = 1/8.

For a predeclared m in {1,2,3,4}, search only the first m blocks and report whether their maximum count is at least 3. The disjoint blocks are independent under the supplied bit law.

The null probability of no exceedance in the search is (7/8)^m. Taking the complement gives p_global(m) = 1−(7/8)^m for this exact independent-block procedure.

Count the same event in the complete outcome space

For one searched block, 512 of the 4096 full outcomes contain an exceedance. The unsearched bits remain free and must still be included when using the twelve-bit outcome space.

For four searched blocks, each block has seven non-exceeding three-bit states. Thus 7⁴ = 2401 full outcomes have no exceedance, leaving 4096−2401 = 1695 with at least one.

The global probability for all four blocks is 1695/4096 ≈ 0.413818. It is larger than the local 1/8 because the search procedure offers additional independent ways to meet the threshold.

This is an exact discrete null-tail calculation. It gives neither the probability that the null is false nor posterior odds for any alternative physical explanation.

Repeating the same block adds no independent opportunity

The product formula used independence of the disjoint blocks. If the same first block is inspected four times, the union event is still just that first block having three heads. Its probability remains 1/8.

The general union bound P(any exceedance)≤Σ_jP(exceedance_j) does not require independence. For m events each having local probability 1/8, it gives the bound m/8 in the table.

A bound need not be attained, and it does not replace the exact search law when that law is available. For four independent blocks the exact probability 0.413818 is below the bound 0.5.

Real overlapping or adaptively chosen tests need their actual dependence and selection rules represented. The repeated-block example does not assign an effective number of trials to a sky survey.

A surprising pattern still needs a physical comparison

A null-tail probability is conditional on the null law, statistic, threshold and search procedure. Changing any of those ingredients changes the question being asked.

This experiment uses a discrete threshold. Its tail probabilities do not acquire a continuous uniform distribution merely because they are called p-values, and no Gaussian-equivalent significance is defined here.

Comparing physical hypotheses requires specified alternatives and their predictions, with an appropriate treatment of uncertainty and prior information where posterior odds are claimed. No alternative likelihood or prior model probability is supplied in this bit experiment.

For a proposed cosmic anomaly, report the searched choices, correlations, calibration and selection history alongside the physical prediction. An unexpected pattern becomes useful evidence through a reproducible comparison, not through its visual appeal alone.

Worked reasoning

Try the reasoning

1. Find the global probability for the first two blocks.

   Show the answer and reasoning

   15/64 = 0.234375.

   The probability of no exceedance is (7/8)² = 49/64; subtract it from 1.

2. What is the exceedance probability if the same first block is inspected four times?

   Show the answer and reasoning

   1/8.

   All four reported events are identical, so their union is the same single event.

3. Does p_global give the probability that the null hypothesis is false?

   Show the answer and reasoning

   No.

   It is a tail probability conditional on the supplied null and search procedure; posterior model odds need additional hypotheses and information.

What to carry forward

• A global probability must describe the complete search that produced the statistic.
• Dependence determines how additional tests change the exceedance probability.
• A null-tail probability alone does not identify a physical cause or posterior model odds.