Effect size, uncertainty and practical importance

The estimate is not the whole finding

A hypothetical study estimates a three-point advantage on a twenty-point delayed test. Another sample or another measurement might give a different estimate. An uncertainty interval represents part of that sampling uncertainty under stated assumptions; it does not include every possible design flaw.

If a reported interval runs from minus one to seven points, the data and model leave room for a small disadvantage as well as a useful advantage. Saying “the method works by three points” would imply more certainty than the interval supports.

Read confidence correctly

A conventional 95 percent confidence interval comes from a procedure designed to include the population parameter in about 95 percent of repetitions under its assumptions. It is not, in the ordinary frequentist interpretation, a statement that a fixed parameter has a 95 percent probability of lying inside this particular interval.

Do not interpret an interval containing zero as proof of no effect. Nor does excluding zero establish a large or valuable benefit. The width, effect scale, design and possible biases all matter.

Standardisation has a purpose and a limit

A standardised mean difference expresses a mean difference relative to a chosen measure of variation. In a simplified constructed illustration, a four-point difference divided by an eight-point standard deviation is 0.5. The unitless value helps compare some outcomes, but its meaning still depends on the test, population and denominator.

Do not turn a conventional label such as small or large into a decision rule. A modest gain on an important capability may matter; a larger score gain may have limited value if it disappears quickly or requires an impractical amount of time.

Join evidence to a decision

State a meaningful outcome before seeing the number. Consider the time, expense, access and opportunity costs of adopting the arrangement. A cheap routine with uncertain small benefits can call for a different decision from an expensive programme with the same estimate.

A p-value does not give the probability that a hypothesis is true, the size of an effect or its practical importance. Read it, when reported, as one component of a broader argument rather than a stamp that ends the interpretation.

Worked example

Try it yourself

Think through each question before opening the answer.

1. Does p = 0.03 mean a 97% probability that a method works?

   Show the answer and reasoning

   No. It does not assign a probability to the hypothesis or establish practical usefulness. Its interpretation depends on the statistical model and analysis.

   The p-value concerns incompatibility with a specified statistical model under assumptions. It is not the probability that a method works, that the hypothesis is true or that the result arose by chance alone.

2. What does a very wide interval suggest?

   Show the answer and reasoning

   The estimate is imprecise under the stated analysis. More informative data or a better measurement arrangement may be needed; selecting only the favourable endpoint is misleading.

   The whole interval expresses imprecision under a model. Selecting its most favourable end as the likely benefit ignores other compatible values and the design’s separate bias questions.

What to carry forward

• Read the estimate and its uncertainty together.
• Statistical significance does not measure practical importance.
• Use a decision-relevant scale and include costs.