Experimental design, factors, controls and confounding

Choose the response, factors and experimental units

A response is what the study measures, such as completion time under a defined task. A factor is an input or condition being compared; its levels are the specified alternatives. Identify the experimental unit that independently receives a condition. Many repeated readings of one treated item are not automatically many independent treatment applications.

A confound occurs when effects cannot be distinguished through the design. If every old-layout run uses batch A and every new-layout run uses batch B, a difference can reflect layout, batch or both. More readings within those unchanged groups do not separate the two effects. Plan the allocation and sequence before examining the outcomes.

Block, randomise and replicate with a reason

Blocking groups runs under a relevant shared nuisance condition, such as a material batch, while comparing the factor levels within that group. Randomising an appropriate run order helps prevent a chosen condition from systematically coinciding with another changing influence. Randomisation does not guarantee perfect balance in a small experiment or remove every unrecorded problem.

Independent replication provides information about variation among experimental applications. Record what is held fixed and what changes between replications. Repeat measurements can additionally reveal measurement variation, but they answer a different question. If practical constraints prevent a simple randomised allocation, preserve those constraints and use a design and analysis appropriate to them.

Write down what receives each condition

Make the old/new layout and basic/clear guidance comparison concrete. In this original proposed study, an experimental unit is one person’s single timed task with a fresh kit. A person receives just one layout/guidance combination. There are two kit-material batches M1/M2, each supplying eight kits. Recruit eight intended users for each batch block, provide the same relevant familiarisation, then randomly assign the four combinations to two people per combination within each block. IDs in the table label one possible allocation outcome, not a completed participant study.

One possible randomised sequence is M1: P04, P01, P07, P05, P02, P08, P03, P06; M2: P14, P10, P15, P12, P09, P16, P11, P13. Retain the actual assignment procedure and sequence in a real study. Every combination occurs twice in each batch, so layout/guidance comparisons can be made within batch rather than substituting one material batch for another. Random assignment addresses systematic allocation; it does not make all people identical or guarantee a balanced outcome in a small study.

There are 2 blocks×4 combinations×2 distinct person-task applications=16 experimental units. Reading each task’s video timing twice produces 32 timing readings, not 32 independent applications of the conditions. Record completion, errors and assistance as well as time. Responses from different applications supply replication information; rereading the same event principally checks the measurement.

The one-condition-per-person choice matters because seeing clear guidance may teach a solution that cannot be removed by switching back to the basic card. Randomising a crossover order would not erase that learning. Using different people avoids that particular within-person carryover but introduces between-person variation. Where people or shared sessions introduce dependencies, retain those identities and use a suitable design/analysis; the table alone does not prove independence or representativeness. It is an allocation plan, not a new source of results for the earlier supplied cell means.

One factor can change another factor’s effect

A full factorial design includes all combinations of the selected factor levels. Our original two-factor exercise supplies mean task times for old/new layouts under basic/clear guidance: 20, 18, 18 and 10 seconds respectively. Under basic guidance, the layout change is−2 seconds; under clear guidance it is−8 seconds.

The difference between those changes is−6 seconds, a descriptive interaction. There is no single layout effect that fully describes both guidance conditions. The plot’s nonparallel lines make the dependence visible. Cell means alone do not provide the replicate variation or model checks needed for an inferential claim; these invented numbers are chosen to teach the comparison.

Repair a confounded proposal

Instead of testing only old layout/batch A and new layout/batch B, a possible study includes both layouts within each available batch and randomises an appropriate sequence within the batch. This creates within-batch comparisons. It does not justify ignoring learning, carryover, different workers or other relevant conditions.

For the supplied factorial means, average old-layout time is (20+18)/2=19 seconds and new-layout time is (18+10)/2=14 seconds. The averaged change−5 seconds summarises the balanced two-guidance account, but hides the separate−2 and−8 changes. Report the interaction and intended conditions instead of claiming every task will improve by five seconds.

Try the reasoning

1. Can more readings resolve a design where treatment and batch always change together?

   Show the answer and reasoning

   No. Repetition within the same confounding pattern cannot identify the separate effects.

2. Why distinguish independent replication from rereading the same item?

   Show the answer and reasoning

   They reveal different variation and support different claims about the treatment and measurement process.

3. What do the nonparallel lines show here?

   Show the answer and reasoning

   The descriptive layout difference depends on the guidance level in the supplied cell means; statistical uncertainty remains unspecified.

4. A proposal compares old layout using novice people with batch M1 against new layout using experienced people with M2. You have 16 people, eight of each experience group, and kits from both batches. With guidance held fixed, how could you repair the comparison?

   Show the answer and reasoning

   Form four blocks: novice/M1, novice/M2, experienced/M1 and experienced/M2, allocating four people to each block and randomly assigning two to each layout within it. Each person performs one task on a fresh kit; randomise an appropriate within-block sequence. This gives both layouts under every stated experience/batch condition and 16 person-task units. Record task and measurement conditions. It supports the planned comparisons under these conditions, not a claim about every material batch or user population.

5. Does obtaining five timing readings from each of four task videos create 20 independent experimental applications? How does the guidance carryover concern affect a proposed repair?

   Show the answer and reasoning

   No: there are four task applications and 20 readings of those events. To add independent treatment applications, assign conditions to additional appropriate person-task units. Reusing the same people across guidance levels changes the design: later performance may retain learning from the clear guidance. Random order alone cannot remove that mechanism. A between-person comparison avoids that carryover at the cost of needing enough people and accounting for their variation.

What to carry forward

Design comparisons so that consequential effects can be distinguished, and examine combinations when one factor can change another’s effect.