Regression, parameter estimation, and model selection

Specify quantities and the model family

A regression model relates supplied inputs to a numerical response. For this lesson, x is a position coordinate in metres along a stated fixed axis, and y is voltage relative to a fixed electrical reference. The original teaching data are exact stipulated numbers, not readings from an assessed instrument. We first choose the candidate relation ŷ(x) = a + b x on 0 ≤ x ≤ 3 m, where a has units V and b has units V/m.

The parameters are the adjustable constants of the chosen relation. A straight line uses two, a and b. A relation ŷ(x) = a + b x + c x² curves in x but remains linear in the unknown parameters a, b and c; c has units V/m². Squaring an input is different from multiplying or otherwise nonlinearly transforming the unknown parameters.

The choice of model family and fitting criterion is part of the task. A fitted relation may summarize data or predict a response within a stated use, but fitting alone supplies no mechanism, intervention result or proof that the relation holds outside the examined conditions. Keep the data's quantities, domains and provenance with the model.

Define residuals and the fitting criterion

For n supplied pairs (x_i, y_i), define the signed residual r_i = y_i − ŷ(x_i): observed response minus fitted prediction. Its unit is V. Ordinary least squares, or OLS, chooses the parameters to minimize Q = Σ r_i², the sum over all fitted observations.

For the same fixed dataset, minimizing Q and minimizing Q/n give the same parameters because n is a positive constant. Q and the training mean squared error Q/n both have units V². Neither is a voltage, a parameter's uncertainty or evidence that residuals are normally distributed. A residual is a difference from the fitted model; it need not equal an unknown instrument error or the model's unobserved random error.

Denote the fitted line parameters by a_fit and b_fit. When S_xx = Σ(x_i − x̄)² is positive, the line's formulas are b_fit = S_xy/S_xx and a_fit = ȳ − b_fit x̄, where S_xy = Σ(x_i − x̄)(y_i − ȳ). S_xx has units m² and S_xy has units m·V, so the slope has units V/m. The positivity condition matters: a formula with a zero denominator does not identify a slope.

Fit the original four-point line

Supply these four training pairs only: (0 m, 1 V), (1 m, 2 V), (2 m, 2 V) and (3 m, 5 V). Give every squared residual equal weight. The means are x̄ = 1.5 m and ȳ = 2.5 V; the centered sums are S_xx = 5 m² and S_xy = 6 m·V. Therefore b_fit = 1.2 V/m and a_fit = 0.7 V.

The fitted predictions are 0.7 V, 1.9 V, 3.1 V and 4.3 V. Subtracting them from the supplied outputs gives residuals +0.3 V, +0.1 V, −1.1 V and +0.7 V. They sum to zero, and Σ x_i r_i = 0 m·V. Their squared sum is Q = 1.8 V², giving training mean squared error 0.45 V².

These checks concern the exact OLS problem. They do not say the residuals are independent random errors or that the slope is a measured physical constant. The worked example also verifies the global minimum rather than relying only on a stationary-point calculation. No Gaussian assumption is needed to find this deterministic minimizer.

Check parameter identifiability

Identifiability asks whether the supplied inputs distinguish different parameter choices in the chosen family. For a line, at least two distinct input positions distinguish its intercept and slope: if δa + δb x is zero at both positions, subtraction forces δb = 0 and then δa = 0. In matrix language, the corresponding model columns have full column rank.

For a separate original case, place every observation at x = 2 m and supply y = 3 V, 4 V and 5 V. Every prediction then depends only on c = a + (2 m)b. The least-squares choice is c = 4 V, with residuals −1 V, 0 V and +1 V and Q = 2 V². The data cannot distinguish a = 4 V, b = 0 V/m from a = 0 V, b = 2 V/m, or any other pair giving the same c.

More repeats at that same position can inform the combined response there, but cannot create the missing distinction between a and b. Here S_xx = 0 m². At a different position, the indistinguishable parameter pairs can give different predictions; selecting one pair through an extra constraint is an additional modeling choice. Full rank establishes uniqueness for this least-squares parameter problem, not the truth of a physical model.

Compare frozen models on held-out points

Training fit uses the observations that determine the parameters. A held-out comparison evaluates fixed predictions on observations not used to fit those parameters. For this supplied comparison, fix two candidate families, their training rule and the validation points before comparing scores; do not refit either candidate using the validation outputs.

The first frozen candidate is the fitted line. The second is the cubic interpolating the same four training pairs: write z = x/(1 m), then ŷ_C(x) = (1 V)[1 + (17/6)z − (5/2)z² + (2/3)z³]. The normalized z is dimensionless, and all four coefficients represent voltage contributions. The cubic's training residuals are zero. Compare both on the stipulated validation pairs (0.5 m, 1.5 V) and (2.5 m, 3.5 V), which lie inside the training input range.

The line predicts 1.3 V and 3.7 V, giving validation mean squared error 0.04 V². The cubic predicts 15/8 V and 23/8 V, giving 17/64 V² = 0.265625 V². The line therefore wins this specified two-candidate validation comparison despite its larger training error. This result establishes the scores on these two points, not universal future performance. Once their outputs guide selection, they are validation data rather than an untouched final test.

Separate residual spread from uncertainty

Statistical interpretation needs more than minimization. One sufficient model for the usual residual-variance estimate assumes fixed error-free inputs, a correctly specified linear mean, and independent mean-zero response errors with the same finite variance σ². With full column rank, p fitted parameters and n > p, s² = Q/(n − p) estimates that common variance. Finding the OLS minimum itself requires no probability distribution.

For the original line, the arithmetic gives s² = (1.8 V²)/(4 − 2) = 0.9 V² and s = √(0.9 V²) ≈ 0.948683 V. The supplied table establishes no actual stochastic error process, so these numbers alone cannot establish its population variance, a parameter's standard uncertainty or a confidence interval. A slope's uncertainty would have units V/m and depend on the input geometry as well as the error model. Distribution-specific interval claims need additional justified assumptions.

The interpolating cubic has n = p = 4. Its zero training residuals do not show that observation noise or parameter uncertainty is zero; Q/(n − p) is undefined there. In a rank-deficient family, merely counting parameter labels is also insufficient: the number of independently fitted directions must be established. Keep descriptive fit scores, conditional statistical calculations and actual measurement uncertainty distinct.

State weights and input-error limits

Equal weights are a specified criterion in our original fits. A weighted criterion instead minimizes Q_w = Σ w_i r_i², with supplied positive dimensionless relative weights w_i. When a response-error model justifies unequal variances, relative weights can be chosen proportional to σ_ref²/σ_i², where σ_ref is a common reference scale. Such a choice needs evidence about those variances; NIST also discusses the limitations of estimated weights.

For a small separate weighted exercise, use outputs 1 V and 3 V and fit a constant c with exact weights 4 and 1. The criterion is 4(1 V − c)² + (3 V − c)². Its minimizer is c = 1.4 V, rather than the unweighted mean 2 V. These weights are stipulated choices, not measured standard uncertainties. Multiplying both weights by the same positive factor changes the criterion's scale but leaves its minimizer unchanged.

Our main example treats positions as exact inputs. If the measured positions have relevant error, the task of estimating an underlying physical relation changes; weighting response residuals does not generally resolve input error. Likewise, weighting alone does not establish independence, remove unknown drift or make a wrong model correct. Identify the input and response error models appropriate to the intended use.

Preserve the selection and physical scope

Keep the data split, quantities and units, candidate families, fitted parameters, transformations, fitting criterion and held-out scoring rule in the record. Scaling or other transformations learned from data belong in the fitting procedure. If validation outputs are repeatedly used to change candidates, a final performance claim requires separate evaluation rather than relabeling those same outputs as untouched test data.

A positive fitted slope describes the chosen relation in the supplied data. It does not show that intervening on position would cause that voltage change, or that the fitted polynomial is a physical law. Such conclusions need the mechanism, experimental design and conditions relevant to the question. The preceding lesson's controls and design remain separate evidence; prediction scores cannot substitute for them.

The three exact cases teach fitting, identifiability and a bounded model comparison. They establish no real sensor calibration, Gaussian error law or universal ranking of straight lines and cubics. The next subject, Laboratory safety, reproducibility, and replication, connects experimental work to its conditions and repeatability; retaining a complete modeling record helps others inspect and reproduce an analysis.

Worked examples

Try it yourself

1. Express the original four positions as 0, 100, 200 and 300 cm, with the same outputs. Give the fitted intercept and slope in V/cm, and predict the output at 50 cm. Should the fitted voltages and residuals change merely because the position unit changed?

   Show the answer and reasoning

   The intercept is 0.7 V and slope is 0.012 V/cm. At 50 cm the prediction is 1.3 V. The physical predictions and voltage residuals stay the same.

   One metre is 100 cm, so 1.2 V/m is 0.012 V/cm. Multiplying that slope by 50 cm gives 0.6 V above the intercept. Input units change the slope's numerical representation, not the fitted physical relation.

2. For the original line, use Q = 1.8 V², n = 4 and p = 2. Calculate Q/n, Q/(n − p) and √[Q/(n − p)] with units. Does arithmetic on this stipulated table establish a Gaussian error law or a confidence interval for the slope?

   Show the answer and reasoning

   Q/n = 0.45 V², Q/(n − p) = 0.9 V² and the square root is approximately 0.948683 V. These values alone establish neither a Gaussian error law nor a slope confidence interval.

   The first denominator defines the training mean squared score; the second is the residual-spread convention under a full-rank statistical model with positive residual degrees of freedom. Interpreting the latter statistically and assessing a slope's uncertainty need justified error assumptions and input geometry. The slope's uncertainty would have units V/m.

3. Keep the repeated-position pairs (2 m, 3 V), (2 m, 4 V), (2 m, 5 V), and add (0 m, 1 V). Use equal-weight OLS in the line family. Find the fitted intercept, slope and Q. Does the new input distinguish the two parameters?

   Show the answer and reasoning

   The fitted intercept is 1 V, slope is 1.5 V/m and Q = 2 V². The distinct input now identifies a unique intercept and slope for this criterion.

   Write c = a + (2 m)b. The criterion becomes (1 V − a)² + 2 V² + 3(c − 4 V)², minimized at a = 1 V and c = 4 V. Hence b = (4 V − 1 V)/(2 m). Two distinct input positions restore the line's full column rank; they do not prove a physical law.

4. For a separate specified validation comparison, retain both original frozen fits and the two original validation pairs, then add (1.5 m, 2.5 V). Calculate each new validation mean squared error over the three points. Keep all fitted coefficients unchanged.

   Show the answer and reasoning

   The line's new score is 2/75 V². The cubic's is 59/192 V². The line still has the smaller score on this specified three-point record.

   At 1.5 m the line predicts 2.5 V, adding zero squared residual to its original 0.08 V² sum. The cubic predicts 15/8 V, adding (5/8 V)² = 25/64 V² to its original 17/32 V² sum. Divide each total by three; the candidates are evaluated, not refitted.

5. Fit a constant c to outputs 1 V and 3 V with positive dimensionless weights 4 and 1. Find the minimizer and Q_w there. What happens to the minimizer if both weights are multiplied by nine? No uncertainty model is supplied for the weights.

   Show the answer and reasoning

   c_fit = 7/5 V = 1.4 V and Q_w,min = 16/5 V² = 3.2 V². Multiplying both weights by nine leaves c_fit unchanged and multiplies the minimum criterion by nine.

   The derivative condition gives c = (4 × 1 V + 1 × 3 V)/5. The residuals are −0.4 V and +1.6 V, so Q_w = 4(0.16 V²) + 2.56 V². A common positive factor does not alter which c minimizes the criterion; stipulated weights establish no measured uncertainties.

6. An analyst changes candidate formulas after inspecting the two original held-out output values, then reports the improved score on those same values as an untouched final test. Is that interpretation supported? Would a lower prediction score alone establish that changing position causes a voltage change?

   Show the answer and reasoning

   Neither interpretation is supported. The reused outputs have influenced selection, so their score is not an untouched final test; lower prediction error alone establishes no causal effect.

   An independent final assessment must account for the data used to choose the model and the conditions under which predictions will be made. A fitted predictive relation does not supply intervention evidence or the physical mechanism needed for a causal claim. Both limits apply even if the reported fitting arithmetic is correct.

What to carry forward

• Regression requires a declared model family, quantities and units, a fitting criterion and an intended use.
• The original equal-weight line has intercept 0.7 V, slope 1.2 V/m and squared residual sum 1.8 V²; residuals are observed minus predicted voltages.
• Full column rank distinguishes fitted parameters. Repeats at x = 2 m identify only a + (2 m)b, not a and b separately.
• The original cubic fits training data exactly, while the line has lower error on the two specified validation points; that comparison is bounded to the stated record.
• Q/n is a training score. Q/(n − p) requires a full-rank statistical interpretation with n > p to estimate common error variance; it supplies no automatic uncertainty or Gaussian inference.
• Weights and noisy predictors need explicit models. Preserve selection and evaluation roles, and keep predictive fit separate from causal evidence.