Engineering & Technology · Quality Engineering, Experiments & Process Evidence

Effect sizes, inference, model assumptions and evidence limits

A new layout appears to save two seconds per task. A decision also needs to know how variable the comparison is, what population it represents and whether two seconds would matter. An estimated effect and a yes-or-no significance label cannot answer all of those questions.

Review measurement and uncertainty first. Means, ranges, square roots and percentages are used directly. The inference lesson defines its paired-difference model and supplies its Student-t quantile; mathematics develops probability and inference further. These examples teach interpretation rather than prescribe production sampling, monitoring or release rules.

Back to the chapter

After reading, you can

Interpret an effect estimate and uncertainty in relation to a practical engineering question and model assumptions.

Define the effect before calculating it

An effect size states the difference or relationship of interest in meaningful units. For matched tasks, define saving as old time minus new time, so a positive difference means a shorter new task. Preserve each match and the allocation conditions. Pairing follows a real relationship between observations; matching rows arbitrarily can create a misleading analysis.

Our original five paired savings are 1, 2, 2, 3 and 2 seconds. Their mean is 2 seconds. The sample variance of the differences is [(−1)²+0²+0²+1²+0²]/4=0.5 s². The standard deviation is about 0.7071 s, while the standard error of the mean is 0.7071/√5≈0.3162 s. These last two quantities describe different variation.

An interval has assumptions and a target

For this calculation only, assume the paired differences are independent observations from the same normal distribution. With five pairs there are four degrees of freedom. A two-sided 95% Student-t interval uses the supplied table quantile 2.776, giving 2±2.776(0.3162), approximately [1.122, 2.878] seconds for the mean saving.

The 95% level describes the interval procedure’s repeated-sampling coverage under its assumptions. It is not a 95% probability that this fixed mean lies inside the particular realised interval, nor a statement that 95% of individual future tasks will save between those endpoints. The small constructed dataset cannot itself establish the assumed distribution or representativeness.

Five constructed paired differences have mean 2 seconds, sample variance 0.5 s², sd 0.7071 s and SE 0.3162 s. Using t 0.975, 4=2.776 gives 2±0.8779 seconds. Zero lies outside, while supplied useful minimum 1.5 seconds lies inside. The interval does not establish that the mean exceeds 1.5 seconds under this stated two-sided procedure.

Practical importance needs its own criterion

Zero lies outside this interval, but suppose the decision requires a mean saving greater than 1.5 seconds. That supplied minimum lies inside. Under this stated interval-based rule, the result does not establish that the mean exceeds the useful minimum, even though the observed mean is larger. An effect can differ from zero without meeting a practical requirement.

A statistical model cannot repair biased allocation, confounding, missing outcomes or inappropriate measurement. Inspect records, conditions and plausible departures from assumptions. Report the effect, units, interval method, sample definition and limitations. If several outcomes or many alternative analyses were examined, preserve that history rather than presenting only the most favourable result as a preplanned finding.

Compare two different claims

The sum of savings is 10 seconds over five pairs, so the mean is 2 seconds. The interval’s lower endpoint 1.122 is above zero and below the supplied useful minimum 1.5. Thus the same numerical result supports different answers to “different from zero?” and “established above the minimum?” under the chosen procedure.

A decision might also depend on rework, learning time, cost or a different task mix. Those outcomes are absent from the five differences. Do not infer them from a narrow time result. A follow-up should address the uncertainty or missing condition that could change the decision, using an appropriate design and sufficient relevant information.

Try the reasoning

Is the sample standard deviation the same as the standard error of the mean?
Show the answer and reasoning
No. The former describes spread of differences; the latter describes uncertainty in estimating their mean under the stated model.
Does a result inconsistent with zero establish engineering usefulness?
Show the answer and reasoning
No. Usefulness requires a relevant effect scale, decision criterion and other consequential outcomes.
Does a 95% mean interval describe 95% of individual future task savings?
Show the answer and reasoning
No. Individual prediction is a different question with a different uncertainty account.

What to carry forward

Report the size and uncertainty of the effect in its actual scope, then compare it with the engineering decision rather than a significance label alone.

References

Teaching resources and research supporting this lesson’s explanations and examples.

National Institute of Standards and Technology (NIST)4 sources