Populations, samples and selection bias

Say whose property is being estimated

A population is the collection of units about which a question is posed. It need not be all people in a country, and it need not be a collection of people at all. The target might be all occupied dwellings in a city on a stated date, all parts produced during a specified shift, or all river reaches in a named drainage area during a season. Unit, eligibility, place and reference time determine membership. A sample can be large and carefully measured while answering a different population question from the one a reader intended.

An estimand is the target quantity defined by that question. For a finite target population U={1,…,N}, with fixed numerical values y₁,…,y_N, a total is T=Σᵢ∈U yᵢ and a mean is μ=T/N when N>0. If yᵢ is an indicator equal to 1 for a defined property and 0 otherwise, μ is a population proportion. A count of events divided by total exposure defines a different estimand from an unweighted average of unit-specific rates. State the numerator, denominator, units and reference period together.

Sampling units, analysis units and reporting units can differ. A survey may first select schools, analyze students within selected schools and obtain some information from parents. A parent who reports for two children is not thereby two independently selected parents. Likewise, a factory inspection might select containers but report properties of items. A sampling description must explain the link between stages and between the units selected and the units whose outcomes enter the estimand.

This lesson uses proportions and elementary algebra for the core examples. Later sections introduce optional finite expectation, design variance and causal connections explicitly, without assuming calculus or the preceding differential-equation branch. Values in a finite population are treated as fixed for the design-based calculations: the repeated variation comes from the declared sampling mechanism. Other probability models can also be useful, but their assumptions must be stated rather than inferred from a spreadsheet.

A list used for selection is not automatically the target population

A sampling frame is the operational information from which units can be selected or located. It may be a registry, address list, map of areas or a sequence of production times. The frame and the target are conceptually different. New dwellings can be absent from an old address list; closed businesses can remain on a registry; a person can appear through several accounts. An apparently complete file can therefore have undercoverage, overcoverage or multiplicity even before any sample is drawn.

To connect a frame to a target, specify how frame entries correspond to eligible analysis units. If one person can be reached through two independently selectable phone records, that person may have a different inclusion probability from someone with one record. Counting records as people can change both the population size and the weights. Duplicate removal or multiplicity adjustments need reliable identity and linkage rules; silently assuming every row is one distinct target unit does not solve the problem.

The selected sample consists of the units chosen by the design. The observed or responding set consists of units for which the required usable information is obtained. These sets need not coincide. Some selected units may be ineligible, unreachable, unwilling to participate, or missing only a particular item. A final analysis can impose further eligibility or completeness rules. Each stage should be visible because exclusion can be associated with the outcome even when the original selection was random.

For a simple picture, imagine a target of 100 eligible units, a frame containing 80 of them, a random selection of 40 frame units and 30 completed observations. The numbers 100, 80, 40 and 30 answer different questions. The fraction 30/40 concerns response among selected units under an appropriate denominator; it is not the fraction of the whole target that had a chance to enter the frame. This schematic does not assume equal probabilities or identify a correction from the four counts alone.

Specify the mechanism, not just a claim of randomness

A sampling design assigns probabilities to possible selected samples. For a target unit i, the first-order inclusion probability πᵢ is the probability that i appears at least once in the selected sample under that design. In the finite without-replacement examples below, each selected unit appears once. For two units i and j, πᵢⱼ is the probability that both are selected; πᵢᵢ=πᵢ. A probability sample uses a known random selection mechanism with positive inclusion probabilities for the target units to which design-based inference is intended to apply.

A statement such as ‘we asked whoever was available’ does not supply those design probabilities. Volunteering, an open website poll, approaching convenient passers-by and selecting conspicuous cases may produce useful observations, but they require separate assumptions to support a population estimate. Calling a procedure random because the analyst did not consciously choose each person does not determine a probability distribution over possible samples. Software randomization also cannot add absent target units to a deficient frame.

For simple random sampling without replacement of a fixed size n from N distinct frame units, every one of the binomial(N,n) subsets has probability 1/binomial(N,n), provided 0≤n≤N. For 0<n≤N, each unit then has πᵢ=n/N. The subset definition is stronger than equal individual probabilities: it specifies how units occur together. Designs with equal πᵢ can have very different joint inclusion probabilities, possible sample compositions and uncertainty.

Neither an unusual realized sample nor apparent demographic balance establishes design bias by itself. Random sampling can occasionally produce an unbalanced sample; a convenient sample can look balanced on a few measured features. The design concerns the probabilities of all possible outcomes of selection. The realized composition is evidence to investigate and report, while a claim about bias needs a target, estimator and mechanism or model. Avoid replacing that analysis with the appearance of one dataset.

Equal inclusion probabilities can hide different possible samples

Consider four labeled units with outcomes y₁=y₂=0 and y₃=y₄=10. Select two units. Under simple random sampling, the six subsets {1,2}, {1,3}, {1,4}, {2,3}, {2,4} and {3,4} each have probability 1/6. Each unit occurs in three subsets, so its inclusion probability is 3/6=1/2. The corresponding sample means are 0, 5, 5, 5, 5 and 10. Enumerating the subsets makes both the equal inclusion and the variation visible.

Now group the units into clusters {1,2} and {3,4}, choose either cluster with probability 1/2 and observe both its units. Each individual still has inclusion probability 1/2, but only two of the six size-two subsets are possible. The sample mean is 0 or 10 with equal probabilities. This is a valid probability design for this finite target, but it is not simple random sampling of two individual units. Its all-or-nothing cluster composition follows from the selection rule.

The target mean is 5. The expected sample mean is also 5 under both designs: under SRS it is (0+5+5+5+5+10)/6, and under the cluster design it is (0+10)/2. Thus the cluster mean is design-unbiased despite every realized cluster sample containing only low or only high outcomes. The more extreme possible estimates are a precision issue here. They do not establish systematic overestimation or underestimation across the declared random selections.

The joint probabilities explain what the marginals miss. Under SRS, any two distinct units are selected together with probability 1/6. Under the cluster design, a same-cluster pair has probability 1/2 and a cross-cluster pair has probability 0. These probabilities determine covariation of inclusion indicators. When designing uncertainty calculations, preserve the actual group structure rather than treating all observed rows as independently selected individuals.

Use the structure of selection when comparing designs

Stratification divides the frame into nonoverlapping groups and samples within each group according to a declared design. A stratum might be a geographic region, production line or other auxiliary category known before observing the study outcome. In the four-unit example, use {1,2} as one stratum and {3,4} as the other, then select one unit uniformly from each, independently across strata. Every possible sample contains one value 0 and one value 10, so its mean is exactly 5. The lack of variation depends on these constant within-stratum outcomes.

Cluster sampling instead selects groups and observes all or a sample of their members. It can reduce travel or administrative effort because selected units are nearby or share a sampling list. When group members resemble each other in the outcome of interest, observing more members of one group can provide less new information than observing similarly many units spread over groups. That is an explanation to examine for a given outcome and design, not a universal theorem that clustering must always be worse.

Multistage designs can combine these ideas: select areas, then households within selected areas, then a person within selected households. Probabilities at later stages are conditional on the earlier selections. The final inclusion probability follows the actual selection pathways and their conditional probabilities. If a unit can be reached by more than one pathway, simply multiplying one path’s probabilities can miss the other ways of inclusion. The worked product later in the lesson explicitly assumes one nested route.

Systematic selection uses a random start and a fixed interval on an ordered frame under a specified rule. It is not automatically the same as simple random sampling: ordering and periodic patterns can affect which groups of units appear together. Likewise, sampling with replacement produces draw probabilities and possible repeated observations that must be distinguished from probabilities of inclusion at least once. Choose the estimator and uncertainty calculation for the actual design rather than transfer a familiar formula solely because the sample size is the same.

Inverse inclusion weights estimate a total under explicit conditions

If unit i has known πᵢ>0 and contributes its value yᵢ when selected, its design weight is wᵢ=1/πᵢ. The Horvitz–Thompson total is T̂_HT=Σᵢ∈s yᵢ/πᵢ, where s is the selected set. A unit selected with probability 1/10 contributes ten times its observed value to this total. The interpretation is expansion under the design; it does not assert that the observed unit is literally identical to ten particular unseen units. A weight is tied to a sampling mechanism and an estimand.

The proof of unbiasedness is finite algebra. Define Iᵢ=1 if unit i is included and 0 otherwise, so E_d(Iᵢ)=πᵢ, where E_d means the probability-weighted average across possible samples under the design. Write T̂_HT=Σᵢ∈U Iᵢyᵢ/πᵢ. By linearity, E_d(T̂_HT)=Σᵢ∈U yᵢ E_d(Iᵢ)/πᵢ=Σᵢ∈U yᵢ=T. No independence of the Iᵢ is used. The proof does require the correct positive inclusion probabilities and the stipulated availability of each selected value.

When the population size N is known, μ̂_HT=T̂_HT/N is design-unbiased for μ=T/N. A different estimator is the ratio μ̂_R=(Σᵢ∈s wᵢyᵢ)/(Σᵢ∈s wᵢ) for a nonempty sample with positive weights. Its denominator is a sample-dependent estimate of population size unless the design or calibration makes it fixed. The expectation of a ratio is not generally the ratio of expectations. Normalizing weights can be useful, but it changes the estimator and its finite-sample properties.

Weighting cannot be justified merely by selecting numbers that make a desired result appear. Document design weights, any response adjustments, calibration constraints and changes such as trimming separately. A survey inclusion probability, a conditional response probability and a treatment propensity describe different mechanisms. They can enter a carefully derived estimator together, but they are not interchangeable labels for the same inverse weight. If values are missing after selection, the pure design proof alone no longer establishes the claimed observed-data estimator.

A three-unit example separates three estimators

Take the fixed population values 2, 4 and 8, with total 14 and mean 14/3. Select exactly one unit, with probabilities 1/2, 1/3 and 1/6 respectively. These probabilities sum to 1 and are also the units’ first-order inclusion probabilities because there is only one draw. The three possible Horvitz–Thompson totals are 2/(1/2)=4, 4/(1/3)=12 and 8/(1/6)=48. Their probability-weighted average is (1/2)4+(1/3)12+(1/6)48=14.

A naive expansion by N=3 instead gives totals 6, 12 or 24. Its expected value is (1/2)6+(1/3)12+(1/6)24=11, so its design bias for the total is 11−14=−3. The low value 2 is more likely to be selected than the high value 8. Treating the draw as if all three units had probability 1/3 ignores exactly the feature that makes the ordinary expansion wrong for this design.

For the known-size HT mean, divide the correct total estimator by 3; its expectation is 14/3. For the ratio-weighted mean based on this one observation, the weight cancels: (yᵢ/πᵢ)/(1/πᵢ)=yᵢ. Its expectation is (1/2)2+(1/3)4+(1/6)8=11/3, with bias 11/3−14/3=−1. Therefore positive, correct weights in a ratio do not guarantee exact finite-sample unbiasedness. The numerator’s unbiasedness and the denominator’s unbiasedness do not prove it either.

The correct HT total can still be variable. Its squared deviations from 14 are 100, 4 and 1156, so Var_d(T̂_HT)=(1/2)100+(1/3)4+(1/6)1156=244. The units of this variance are squared outcome-total units; the standard deviation is √244=2√61 in total units. Unbiasedness concerns the center over repeated selections, not closeness of every possible estimate or superiority in mean-squared error. This deliberately small example makes those distinctions visible without asymptotic approximations.

Distinguish an error from a property of the estimator

For an estimator θ̂ of a fixed target θ, a realized error is θ̂−θ. Its design bias is E_d(θ̂)−θ, and its design variance is Var_d(θ̂)=E_d[(θ̂−E_dθ̂)²]. The design standard error is √Var_d(θ̂); in practice it often must itself be estimated. When these finite expectations exist, MSE_d(θ̂)=E_d[(θ̂−θ)²]=Var_d(θ̂)+Bias_d(θ̂)². This identity follows by expanding θ̂−θ=(θ̂−E_dθ̂)+(E_dθ̂−θ) and noting that the first term has mean zero.

For the four-unit population, the SRS mean has variance (25+0+0+0+0+25)/6=25/3. The cluster mean has variance (25+25)/2=25. The one-per-stratum mean has variance 0 in this special constant-within-stratum case. All have expectation 5. The cluster-to-SRS variance ratio, called a design effect for this estimator, outcome and comparison, is 25/(25/3)=3. This is not a design effect for every possible variable measured on those units.

The same SRS variance can be written (1−n/N)S²/n, where S²=Σᵢ∈U(yᵢ−μ)²/(N−1), N>1, and 1≤n≤N for this without-replacement mean. Here S²=100/3, N=4 and n=2, giving (1−1/2)(100/3)/2=25/3. The finite-population correction reflects the specified sampling without replacement. The population spread S², variance of an estimator and squared standard error play different roles. This formula should not be applied to clustered, nonresponding or convenient observations just by substituting their row count.

An advanced distinction concerns which uncertainty can be estimated from an observed sample. For a fully specified finite population, Var_d(T̂_HT)=ΣᵢΣⱼ(πᵢⱼ−πᵢπⱼ)yᵢyⱼ/(πᵢπⱼ). This follows from Cov_d(Iᵢ,Iⱼ)=πᵢⱼ−πᵢπⱼ. The usual unbiased sample-based pair formula divides relevant terms by πᵢⱼ and therefore needs positive pair probabilities for those terms. In the one-draw example all πᵢ>0 but πᵢⱼ=0 for distinct units. The total remains unbiased and its theoretical variance is defined, but the ordinary pairwise unbiased variance estimator cannot be formed for the unobserved pairs. Extra structure or conservative alternatives require their own argument.

Equal sample counts need not represent equal population counts

Suppose a population has two known strata: 80 units in A, each with outcome 10, and 20 units in B, each with outcome 50. Select 10 units by SRS within each stratum and obtain every selected outcome. The population total is 80·10+20·50=1800, and its mean is 18. The ordinary mean of the 20 observed rows is (10·10+10·50)/20=30 because the smaller stratum contributes half the sample instead of one fifth of the population.

The stratum inclusion probabilities are 10/80=1/8 and 10/20=1/2, so the design weights are 8 and 2. The weighted total is 10·8·10+10·2·50=1800. The sum of weights is 10·8+10·2=100, the known population size. Here dividing the weighted total by N or by the sum of weights gives the same mean 18. The denominator is fixed by this particular fixed-allocation stratified design, unlike the random denominator of the earlier one-draw ratio example.

The weighted estimate is exact in every possible sample here because all outcomes within each stratum are identical. In an ordinary population with variation within strata, the same known-N HT construction is design-unbiased under the stated within-stratum selection, but individual estimates vary. Disproportionate allocation can deliberately improve information about a small stratum; it is not a mistake merely because sample fractions differ. The error is to ignore the allocation when interpreting an estimator for the whole population.

Calibration adjusts weights to satisfy known auxiliary totals or counts. It can align an estimate with the population’s known stratum composition and can sometimes improve precision under suitable designs and relationships. It does not reveal unobserved outcomes by algebra alone. If respondents within a calibration cell differ systematically in the outcome from nonrespondents, reproducing the cell count does not guarantee that its weighted outcome total is correct. The target, auxiliary information, adjustment rule and assumptions must remain visible.

Response is an additional selection process

Unit nonresponse occurs when a selected eligible unit does not provide the required survey information; item nonresponse concerns a particular missing variable. Both can change the observed set. If people with longer working hours are less likely to respond to a survey about working hours, an initially sound probability sample can yield an unrepresentative respondent mean. The concern is a relationship between the observation process and the target quantity, not simply the existence of missing records.

Consider a target of 100 units in two known auxiliary cells A and B of size 50 each. In this hypothetical population, every A unit has binary outcome 1 and every B unit has outcome 0. Invite everyone. All 50 A units respond, while exactly 10 B respondents are chosen uniformly within B. The response rate is 60/100=3/5; the respondent outcome proportion is 50/60=5/6; the target proportion is 50/100=1/2. Under this fixed-count response design, every possible respondent set has the same mean, so its bias is 5/6−1/2=1/3.

The known auxiliary cells must be available beyond the respondents; they are not labels reconstructed from the missing outcomes. In the specified example, response probabilities are 1 in A and 1/5 in B. Weights 1 and 5 produce weighted outcome total 50 and weighted count 100, recovering proportion 1/2. Exact recovery depends on the constant outcomes within cells and the stated response mechanism. For variable outcomes, a response adjustment needs appropriate assumptions about the relation between response and outcomes within the available cells.

A response rate alone cannot identify the size or direction of nonresponse bias. A low rate could occur under an outcome-independent mechanism, while a higher rate could still exclude a scientifically distinctive group. Report response definitions, follow-up procedures, observed differences and sensitivity to plausible missing-data assumptions. Neither imputing a value nor increasing a respondent’s weight makes an assumption disappear. Missingness mechanisms and their identification receive fuller treatment in later statistics; here the central task is to show where selection enters.

Some target information cannot be obtained from the same frame

Suppose 20 of 100 target units never occur in the sampling frame. Each has inclusion probability 0 under every sample restricted to that frame. If the remaining 80 units all have outcome 0, even observing all of them leaves open what the excluded units would have reported. A larger sample drawn from the same covered group gives more complete information about that group, but it does not create observations from the absent one.

Compare two possible fixed target populations. In world A, all 20 uncovered units also have outcome 0, so the target mean is 0. In world B, all 20 uncovered units have outcome 1, so the target mean is 20/100=1/5. The frame, selection mechanism and every covered outcome are identical in the two worlds. Consequently, no analysis using only those covered data can distinguish these target means without additional information or assumptions. This is an identification limitation, not merely a large sampling standard error.

The HT proof required πᵢ>0. There is no finite design weight 1/πᵢ when πᵢ=0, and multiplying observed outcomes by larger finite weights does not recover an unknown excluded-group outcome. A new frame, supplementary data, a narrower declared target or an explicit extrapolation model might address the problem. Each changes the information or question. Reporting a small variance for the covered-group estimate does not establish that its value is the desired whole-target mean.

For bounded binary outcomes in this particular example, the covered data do give a useful bound: the unknown total lies between 0 and 20, and the target mean lies in [0,1/5]. The two constructed worlds attain the endpoints. Such partial information should not be replaced by a single unjustified point estimate. Bounds, sensitivity analyses and scientific knowledge can clarify what is and is not learned, provided their assumptions and target remain explicit.

More records address some uncertainty but not every source of error

Increasing a well-designed sample can reduce sampling variation, but ‘more observations’ is not a universal cure for selection. In the zero-coverage example, a census of the 80 frame units still leaves the 20 target outcomes unidentified. Repeatedly adding records from an open volunteer mechanism can make the respondent distribution very stable while preserving a difference from the target. The distinction is between precision for an observed mechanism and validity for the intended population quantity.

Under SRS without replacement from the correctly defined finite target, observing all N units makes the sampling variance of its descriptive mean zero. That claim concerns the declared values and the sampling stage. Measurements can still have error, records can be duplicated or mislinked, questions can be interpreted differently, and the chosen definition can fail to capture the intended construct. A nominal census therefore does not establish perfect data quality, and a census of a frame is not automatically a census of the target.

Selection bias is a systematic discrepancy tied to how units enter an analysis and how the chosen estimator relates to its target under a specified mechanism or model. Coverage and nonresponse are common routes. Measurement error, processing error and sampling variation are distinct concepts even though they can interact. It is useful to trace them separately because a design weight aimed at unequal selection does not automatically repair a miscalibrated instrument or a changed outcome definition.

When comparing procedures, define the loss and constraints. An unbiased estimator may have large variance; a deliberately stabilized or modeled estimator may introduce bias while lowering mean-squared error under stated assumptions. Cost, precision, robustness and coverage can all matter. The finite MSE identity describes a mathematical tradeoff, but it does not authorize concealing bias or choosing an estimator after seeing a preferred result. Preserve the design and report sensitivity to assumptions that cannot be checked from observed outcomes alone.

Multiply the right conditional probabilities

Take a unit contained in a single nested cluster. Let C mean that the cluster is selected, S mean that the unit is selected within it, and R mean that the unit supplies the needed response. Suppose P(C)=1/4, P(S|C)=2/5 and P(R|S,C)=4/5. Because selection of the unit requires its cluster, the unit inclusion probability is P(S∩C)=P(C)P(S|C)=1/10. Its design weight for the selected-unit total is 10.

The probability of an observed response through this route is P(C∩S∩R)=P(C)P(S|C)P(R|S,C)=(1/4)(2/5)(4/5)=2/25. If these probabilities are truly known for that unit and the relevant outcomes are observed whenever R occurs, an inverse observed-probability construction uses 25/2. The multiplication is the conditional-probability chain rule. It does not assume that cluster inclusion, unit selection and response are independent events with those marginal probabilities.

In practice response probabilities are often unknown and estimated from available covariates. An estimated response model adds identification, estimation and specification assumptions. The chain-rule arithmetic cannot establish that the model contains the variables needed to make response conditionally unrelated to the outcome. Likewise, multiple routes to selection, unequal cluster sizes or within-household rules must be included in the actual probability calculation. A single illustrative product is not a universal weighting formula for every complex survey.

Keep design and response stages in the record so an analyst can reproduce the reasoning. A selected unit may have a valid design weight even when one outcome is missing. Complete-case analysis then introduces outcome-specific observation probabilities. Different variables can therefore use different observed sets despite originating in the same survey. Documenting these mechanisms is essential for comparisons across outcomes, years and linked datasets; a common filename or a shared sample-size label does not make the estimands identical.

Population description and causal effects ask different questions

Random selection determines who enters a study from a population or frame. Random assignment determines which treatment, intervention or exposure level the study units receive under an experimental design. One can occur without the other. A probability sample observed without an intervention can support a descriptive population estimate under its design. A randomized experiment among volunteers can support an internally valid treatment comparison under appropriate conditions, while its target-population scope requires a separate argument.

For the causal connection, let Yᵢ(1) and Yᵢ(0) denote the outcome unit i would have under two specified exposure levels. Their difference is an individual causal contrast, and a population average contrast is N⁻¹Σᵢ[Yᵢ(1)−Yᵢ(0)]. Only the outcome under the received exposure is normally observed. These potential outcomes are a mathematical way to define a question; specifying them in a thought experiment does not mean they can be recovered from an ordinary census.

Imagine observing all 100 units in a finite target. Fifty have Yᵢ(0)=Yᵢ(1)=1 and fifty have Yᵢ(0)=Yᵢ(1)=0. Expose 40 of the first group and 10 of the second. Among 50 exposed units the observed mean is 40/50=4/5; among 50 unexposed units it is 10/50=1/5. Their difference is 3/5. Yet every individual causal contrast is 0, so the true population average effect in this stipulated world is 0. Perfect descriptive coverage did not remove confounding in exposure assignment.

Observational causal identification is possible under suitable designs and assumptions; it is not ruled out simply because assignment was not randomized. A common sufficient setup requires a well-defined intervention and consistency linking observed and potential outcomes, conditional exchangeability of exposure given adequate pre-exposure covariates, and positive probability of each relevant exposure within those covariate groups. Assumptions about interference and the assignment structure also matter. The point here is to distinguish those causal requirements from survey inclusion probabilities, not to infer a causal effect from a sample weight alone.

Randomization does not automatically transport an effect

An effect can vary across groups. Suppose a target is 20% group A and 80% group B. In a hypothetical world, the average treatment effect is 4 in A and 1 in B, measured in the same outcome units under the same intervention definition. The target average effect is 0.2·4+0.8·1=8/5. A perfectly randomized trial containing only A units can identify an A-specific contrast under the trial’s causal conditions, but its population contrast is 4 in this example rather than 8/5.

The group-B effect 1 is stipulated for this thought experiment, not estimated from the A-only trial. If a different world had another B effect but exactly the same A outcomes and trial records, the trial alone would not distinguish the target effects. Weighting the observed A participants cannot create observed B treatment comparisons. This is a support problem for participation in the target, even if both treatment arms are present within A.

When the required target groups are represented, transport or generalization can sometimes use covariates, target distributions and explicit assumptions about how causal effects or potential outcomes relate across participation groups. A suitable overlap condition, compatible measurement and intervention definitions, internal causal validity, and justified cross-population assumptions are required. The exact sufficient conditions depend on the scientific question and strategy. Treatment positivity within a trial and participation positivity across target groups concern different mechanisms.

These connections also arise when taking a factory experiment from one production line to another, transferring an education intervention across schools, or applying an ecological result across habitats. The units, outcomes, effect modifiers and target conditions must be considered scientifically. An abstract about trial generalizability supports the distinction in scope; the arithmetic above is an original finite thought experiment. It is not a claim that one universal reweighting recipe settles transport or a recommendation about any real intervention.

Make the route from target to conclusion inspectable

A clear sampling account begins with the target population and estimand, then describes the frame, sampling units, stages, eligibility rules, selected counts and usable-response counts. State whether selection was with or without replacement and how randomization was implemented. Report available inclusion probabilities or weights and the procedure used to estimate uncertainty. For a multistage survey, preserve the strata and cluster information needed for the design rather than stripping it away because the final file contains individual rows.

Explain changes between design weights and final analysis weights. If nonresponse adjustment, calibration, trimming or imputation is used, name the auxiliary information, constraints and assumptions. Report the analysis denominator for each outcome and account for missing items and exclusions. These details allow another reader to distinguish a justified population estimate from a respondent summary. A polished chart and a large row count cannot substitute for the selection history.

Match the conclusion to the evidence. A sample mean describes the observed values. A design-based population estimate additionally uses a declared probability design and appropriate estimator. A model-based extrapolation additionally uses its model assumptions. A causal contrast additionally needs an identification strategy, and transfer to another population needs its own scope argument. Each step can be legitimate, but each adds conditions. State uncertainties and limitations at the step where they enter rather than attaching a vague caveat after a broad claim.

The practical habit is to ask four connected questions: Who is the target? How could each unit enter the data? What quantity does this estimator compute under that mechanism? What assumptions connect it to the intended conclusion? The finite examples make the answers checkable: equal inclusion need not mean SRS; an extreme sample need not prove bias; correct ratio weights need not give exact unbiasedness; a census need not identify a causal effect. Carry these distinctions into later probability, estimation, regression and experimental-design lessons.

Worked examples

Try it yourself

1. An online survey claims to describe ‘the city’. Give three pieces of population information needed before a numerical conclusion has a clear target.

   Show the answer and reasoning

   For example: the eligible unit type, the geographic boundary and the reference date or period; the target quantity must also be defined.

   Residents, households, registered businesses and website visitors are different populations. A specified proportion or mean needs both population membership and an outcome definition. The phrase alone does not establish that respondents are a probability sample.

2. A frame contains 400 eligible units. A design selects 100 of them and 70 give usable responses to an item. Is 70/100 a population outcome proportion?

   Show the answer and reasoning

   No. Under the stipulated denominator it is the item’s usable-response fraction, 7/10.

   The numerator counts usable observations rather than units with the outcome of interest. Population description additionally needs the target, coverage, selected-unit probabilities and the response mechanism. The item’s observed set can differ from another item’s.

3. Under SRS without replacement of 3 units from 8 distinct units, what is the probability of each size-three subset and of including one specified unit?

   Show the answer and reasoning

   Each subset has probability 1/56, and a specified unit has inclusion probability 3/8.

   There are binomial(8,3)=56 subsets. A specified unit occurs with any 2 of the other 7 units, hence in binomial(7,2)=21 subsets, giving 21/56=3/8. Equal subset probabilities, not just equal marginal inclusion, define this SRS design.

4. A design selects either {1,2} or {3,4}, each with probability 1/2. Is it SRS of two from four because every unit has inclusion probability 1/2?

   Show the answer and reasoning

   No; four of the six size-two subsets have probability 0.

   SRS would give every size-two subset probability 1/6. The declared cluster design is still a probability design, but its joint inclusion structure differs. Equal individual inclusion probabilities do not determine that structure.

5. A selected unit has measured value 12 kg and known inclusion probability 1/4. What is its contribution to an HT total, and what units does it have?

   Show the answer and reasoning

   Its design weight is 4 and its total contribution is 48 kg.

   The weight is the dimensionless reciprocal of a probability. Multiplying 12 kg by 4 gives 48 kg as an expansion contribution. It does not mean the measured unit itself physically contains 48 kg or identify four particular units with the same value.

6. In the finite HT unbiasedness proof, must the inclusion indicators of different units be independent?

   Show the answer and reasoning

   No. Correct positive first-order inclusion probabilities and linearity of expectation suffice for the total’s design unbiasedness.

   For each unit E_d(Iᵢyᵢ/πᵢ)=yᵢ. Summing these equalities does not require independence. Joint inclusion probabilities matter for variance, and missing outcomes after selection require a separate observation argument.

7. Exactly one selected observation has value y and design weight w>0. What is its normalized weighted mean? Does the correct weight alone prove this mean is population-unbiased?

   Show the answer and reasoning

   The ratio is wy/w=y; the correct weight alone does not prove unbiasedness.

   With one observation the weight cancels. Under unequal selection probabilities the selected value can have an expectation different from the unweighted finite-population mean. A random denominator cannot be replaced by its expectation inside a ratio.

8. An estimator has design expectation 11/3 for a target mean 14/3. Find its design bias. Does a single realized estimate above the target contradict that bias?

   Show the answer and reasoning

   The bias is −1; a realized estimate above the target is compatible with a negative design bias.

   Bias compares the probability-weighted center of all possible estimates with the target. A particular realization can lie on either side. For the one-draw example, the value 8 is above 14/3 even though the ratio mean’s expectation is below it.

9. An unbiased total estimator has design variance 244 in squared kilograms. What are its MSE and standard error, and can its realized error be nonzero?

   Show the answer and reasoning

   MSE is 244 kg²; the design standard error is √244 kg=2√61 kg; its realized error can be nonzero.

   Unbiasedness sets the average signed error to zero, not every error. The standard error is the square root of estimator variance and has the same units as the total. It is not the variance or the spread of individual population values.

10. For the same outcome and sample size, an estimator has cluster-design variance 25 and SRS variance 25/3. Find the design effect and state a limit on interpreting it.

    Show the answer and reasoning

    The variance ratio is 3; it applies to that outcome, estimator, sample size and comparison design.

    Compute 25/(25/3)=3. The result does not by itself compare costs, establish bias, or give the design effect of every other outcome. A particular stratum construction can reduce variance without guaranteeing the same reduction for all variables.

11. A stratum of size 60 is sampled by SRS of 15 with complete response. What is each unit’s inclusion probability and design weight?

    Show the answer and reasoning

    The inclusion probability is 1/4 and the design weight is 4.

    For fixed-size within-stratum SRS, π=n_h/N_h=15/60. The reciprocal expands a selected unit’s contribution for a stratum total. Whole-population estimation must also account for the other strata and the appropriate population denominator.

12. Two surveys each have response rate 60%. Must they have the same nonresponse bias for a population mean?

    Show the answer and reasoning

    No; the response rate alone does not identify the relation between response and the outcome.

    Bias depends on the target, estimator and mechanism. One survey could respond independently of the outcome under a stated model, while another disproportionately misses a distinctive group. The rate is useful process information but is not itself a bias estimate.

13. A target has 200 units with binary outcomes. A complete census of the 150 covered units finds all outcomes 0, and the remaining 50 are uncovered. With no other information, what target-mean bound follows?

    Show the answer and reasoning

    The target mean lies in [0,1/4].

    The unknown group contributes between 0 and 50 positive outcomes. Dividing by the known target size 200 gives the bound. Both endpoints can occur with exactly the same covered records, so no frame-only sample enlargement identifies a unique target mean.

14. A unique nested route has P(C)=1/4, P(S|C)=2/5 and P(R|S,C)=4/5. Is P(C∩S∩R)=2/25 an assumption of mutual independence?

    Show the answer and reasoning

    No; it follows from the conditional-probability chain rule.

    Multiplication of these stated conditional probabilities gives (1/4)(2/5)(4/5)=2/25. Multiplying three unrelated marginal probabilities would instead require justification such as independence. The observed inverse weight is 25/2 only under the declared observation mechanism.

15. A total estimator has variance 41 and bias −3 for a fixed target. Find its MSE and say whether lower MSE than an unbiased competitor would make it unbiased.

    Show the answer and reasoning

    Its MSE is 41+9=50; a lower MSE would not remove its bias.

    The finite identity is MSE=variance+bias². Squared-error performance and unbiasedness are different properties. Comparing them requires a specified target population, mechanism and loss, not an assertion that either property alone settles every scientific choice.

16. In a one-unit sample every πᵢ is positive but every distinct-pair probability πᵢⱼ is 0. Is HT total unbiasedness ruled out, and can the usual unbiased pairwise variance formula simply divide by πᵢⱼ?

    Show the answer and reasoning

    Total unbiasedness is not ruled out; the usual pairwise sample variance formula cannot divide by those zero joint probabilities.

    The point-estimator proof uses first-order positivity. Generic estimation of cross-product variance contributions needs information about pairs; a single draw never observes a distinct pair. A known finite population still has a mathematically defined design variance, and additional assumptions or alternatives require separate arguments.

17. A complete census gives an exposed-minus-unexposed mean difference 3/5. Must this equal the population average causal effect?

    Show the answer and reasoning

    No. Complete coverage does not identify the counterfactual causal contrast or randomize exposure.

    In the stipulated example with Y(1)=Y(0) for every unit, selective exposure creates the association while every causal effect is 0. The descriptive difference is correctly computed; interpreting it causally needs an appropriate design or identification assumptions.

18. A randomized trial contains only A units. The target also contains B units, and the B treatment effect is unknown. Can weights assigned only to A trial participants identify an arbitrary B effect?

    Show the answer and reasoning

    No; the trial lacks B support, and additional information or scientifically justified cross-group assumptions are needed.

    Randomization can justify a treatment comparison within the trial under its own conditions. It does not reveal missing target-group potential outcomes. A stipulated B effect permits a thought-experiment target average, but that stipulation is not an empirical result from the A-only trial.

What to carry forward

• A population question needs defined units, eligibility, place, time, outcome and estimand.
• Target population, sampling frame, selected sample and observed or responding set describe different stages.
• Record identity and multiplicity affect how frame entries correspond to target units.
• A probability design specifies the selection mechanism and the relevant positive inclusion probabilities.
• SRS without replacement makes every fixed-size subset equally likely; equal unit inclusion alone is insufficient.
• Strata, clusters and multistage paths affect joint selection and precision, even at the same sample size.
• The HT total is design-unbiased under correct positive inclusion probabilities and observed selected outcomes; independence is not required.
• A known-size mean and a normalized weighted ratio are different estimators; ratios are not generally exactly unbiased.
• Realized error, design bias, estimator variance, standard error and MSE have different meanings and units.
• First-order positivity can suffice for point unbiasedness while zero pair probabilities obstruct the usual unbiased variance estimator.
• Disproportionate allocation can be intentional; appropriate weights and target denominators are essential.
• Response adjustment and calibration require auxiliary information and assumptions about what remains unobserved.
• Larger frame-only samples cannot identify outcomes of target units with zero coverage; useful bounds may remain possible.
• Conditional multistage probabilities multiply by the chain rule, while estimated response mechanisms add assumptions.
• Random selection, random treatment assignment, complete descriptive coverage and transfer to another population address distinct questions.
• Preserve the selection and adjustment history so population, model and causal conclusions can each be assessed against their conditions.