Inverse problems and Bayesian computation

Start with a position and an unknown velocity

A tracked point is stipulated to follow x(t) = b + vt for 0 ≤ t ≤ 4 s on a fixed straight coordinate line, with right positive. Time t is measured from the declared start; b is the initial position in metres and v is a constant axial velocity in m/s. If b = 1 m and v = 2 m/s, then x(2 s) = 5 m. Does the position 5 m at 2 s identify both b and v when both are unknown?

A forward problem takes specified parameters and predicts an output. An inverse problem uses output data, together with a model, to infer unknown parameters or states. The direction of the calculation changes, while its units, coordinate convention and domain still matter. The positions used here are exact hypothetical inputs; they are not a measurement record or evidence that a real point follows this rule.

Preparation is rearranging a linear expression and multiplying or dividing fractions. Algebra, functions, graphs, and logarithms provides that algebra background. The probability notation below is introduced locally. On a first reading, follow the two candidates and their tables; the finite example can be calculated by hand.

Ask what the data identify

The single equation 5 m = b + v(2 s) has several solutions. For example, b = 3 m with v = 1 m/s and b = 1 m with v = 2 m/s both fit. Identifiability asks whether different allowed parameter choices can give the same supplied outputs. Choosing one pair by an extra preference adds an assumption; it does not add another position observation.

If the same exact model also supplies x(4 s) = 9 m, subtraction gives v = (9 m − 5 m)/(4 s − 2 s) = 2 m/s, followed by b = 1 m. The distinct times make this two-parameter linear solve unique. Repeating the same noiseless equation would not provide that extra constraint.

Uniqueness and sensitivity are different questions. With a known zero initial position and a known positive interval T, the inverse is v = y/T for displacement y. A change Δy gives Δv = Δy/T. An exact stipulated change 0.001 m gives 0.0005 m/s at T = 2 s but 0.05 m/s at T = 0.02 s: a factor of 100. The smaller interval amplifies this input change; these calculations do not assign an instrument uncertainty.

Declare candidates and a reading law

For the following separate model, the initial position is known to be zero and the interval is exactly T = 2 s. The fixed unknown velocity has two allowed candidates: H₁ means v = 1 m/s and H₂ means v = 2 m/s. Their error-free model displacements are 2 m and 4 m. These candidates exhaust this teaching model, rather than all possible velocities of a real system.

Before a reading, assign prior probabilities P(H₁) = 2/3 and P(H₂) = 1/3. P denotes the probability assigned by the model. The velocity stays fixed through the interval; uncertainty about its value does not describe a velocity that randomly changes during the motion. The two prior weights are nonnegative and sum to one.

A displacement readout Y obeys Y = vT + E. Its reading error E can be −2 m, 0 m or +2 m with probabilities 1/4, 1/2 and 1/4, respectively, under either candidate. Thus Y and E have units m, while their event probabilities are dimensionless. The error law and prior are supplied assumptions, not frequencies estimated from an apparatus. Distinct readouts will require an additional assumption about their joint errors.

Multiply, then normalize the weights

Suppose a readout gives Y = 4 m. The likelihood Lᵢ = P(Y = 4 m | Hᵢ) is the probability of this specified reading when candidate Hᵢ is held fixed; the vertical bar means given. H₁ needs E = +2 m, so L₁ = 1/4. H₂ needs E = 0 m, so L₂ = 1/2. The reading is twice as likely under H₂, but likelihood is not yet the probability of H₂ given the reading.

Let πᵢ be the prior weight and wᵢ = πᵢLᵢ its unnormalized updated weight. Bayes' rule gives P(Hᵢ | Y = 4 m) = wᵢ/Z, where Z = w₁ + w₂ is the total probability of this reading under the prior model. It must be positive for this division. Here both weights are 1/6, Z = 1/3, and both posterior probabilities are 1/2.

The posterior is the candidate distribution after conditioning on the data. These finite likelihoods are event probabilities because the reading model is discrete. They need not sum to one across candidates: 1/4 + 1/2 = 3/4 here. The normalized posterior does sum to one. Equal posterior weights result from the stated prior and likelihood together, rather than from equal compatibility alone.

Update with the correct joint model

Now assume distinct readouts of the same interval displacement have independent errors when each candidate is held fixed, each with the supplied error law. This is conditional independence: P(Y₁ = y₁, Y₂ = y₂ | Hᵢ) = P(Y₁ = y₁ | Hᵢ)P(Y₂ = y₂ | Hᵢ). The subscripts label readouts, not later times in the trajectory. Averaging over an unknown shared candidate need not leave the readouts independent.

If both readouts are 4 m, the joint likelihoods are (1/4)² = 1/16 and (1/2)² = 1/4. Multiply by the original prior: the weights are 1/24 and 1/12, summing to 1/8. The posterior becomes P(H₁ | both readings) = 1/3 and P(H₂ | both readings) = 2/3. Alternatively, update the first posterior once with the fresh likelihood; it gives the same result. Do not multiply by the original prior again.

If instead both readouts share exactly the same error, E₂ = E₁, then seeing 4 m twice is the same event as seeing it once. Under that alternative joint model the posterior stays 1/2, 1/2. Treating a duplicate or a perfectly shared error as an independent repeat would manufacture information. Compare the three panels with the table under the declared independent-error model.

Predict a fresh readout

A posterior for velocity and a prediction for a reading answer different questions. After the two readouts, use candidate weights 1/3 and 2/3. For a fresh third readout of the same displacement, assume the same error law and conditional independence from the earlier errors. Its probability at a value y is P(Ynew = y | data) = Σᵢ P(Ynew = y | Hᵢ)P(Hᵢ | data), where Σᵢ means add over the two candidates.

For example, a fresh 4 m reading has probability (1/3)(1/4) + (2/3)(1/2) = 5/12. The full predictive distribution in the table combines uncertainty about the fixed velocity with variation in a new readout. Plugging in only the most probable candidate H₂ would discard the remaining probability assigned to H₁.

A probability-weighted average is a summary: multiply each value by its probability and add. The posterior velocity average is 5/3 m/s, and the predictive reading average is 10/3 m. Neither average is an allowed single value in its respective finite list. In particular, a predicted mean does not describe a guaranteed next reading.

Compute the quantity you need

Finite Bayesian computation can enumerate every allowed candidate: evaluate the reading model, form the weights, check a positive normalizer, divide, and calculate the requested probability or weighted average. The tables provide exact rational check cases for a program. Check nonnegative weights, their normalized sum, units and the conditioning data, rather than checking only that a program returned a number.

A program can also report the most probable candidate: after two readouts it is H₂, with probability 2/3 and velocity 2 m/s. That single selection does not replace the full posterior for predicting new readouts or reporting the remaining candidate uncertainty. Choose a calculation that answers the requested question: a selected candidate, a mean or an event probability.

With a larger or continuous parameter space, sums or weighted integrals may need numerical approximation. Posterior draws can approximate requested averages and predictive probabilities. Such computation introduces approximation or sampling error in addition to uncertainty represented by the model. The present finite calculation is an exact check case; it makes no claim to have run a sampler or diagnosed a general Bayesian algorithm.

Check the assumptions behind the answer

The posterior depends on the supplied prior and reading law. If the priors were instead 1/2 and 1/2, the same single 4 m reading would give posterior weights 1/3 and 2/3. Compare plausible assumptions and explain the reasons for a prior before interpreting the resulting numbers. A prior can express extra information or preference, but it cannot turn indistinguishable data fits into new observations.

A stipulated 3 m readout has probability zero under both candidates and this exact discrete error law. Its normalizer is zero, so the finite Bayes division is undefined. Check the candidate set, the reading-error model and the data encoding. This incompatibility alone does not identify which assumption would fail in an actual experiment.

For an application, inspect what the prior and fitted model predict about relevant readout features and compare them with appropriate data. Also check the coordinate convention, interval, noise dependence and numerical calculation. A normalized posterior is conditional on the chosen model; it does not by itself establish a physical law, a calibrated instrument or the adequacy of the two-candidate restriction.

Worked examples

Try it yourself

1. For a known zero initial position and a constant-velocity model, a stipulated displacement is 3 m at T = 1.5 s. Find v. Why would a datum at T = 0 s not support the same division?

   Show the answer and reasoning

   v = 2 m/s. At T = 0 s the division is undefined and the model displacement is zero for every velocity.

   For the positive interval, v = (3 m)/(1.5 s) = 2 m/s. At zero elapsed time the equation y = vT cannot distinguish velocities; a nonzero displacement would instead be inconsistent with that model at its known zero initial position.

2. Keep the two candidates, the 2 s interval, the supplied error law and one 4 m readout, but assign equal prior weights 1/2. Find the posterior probability of H₂.

   Show the answer and reasoning

   P(H₂ | Y = 4 m) = 2/3.

   The weights become (1/2)(1/4) = 1/8 and (1/2)(1/2) = 1/4. Their sum is 3/8, so the second posterior is (1/4)/(3/8) = 2/3. Changing the prior changes this conditional answer.

3. Start from the main two-reading posterior 1/3, 2/3. A fresh conditionally independent readout is also 4 m. Find the new posterior probability of H₂. Use the existing posterior as the starting weights.

   Show the answer and reasoning

   The new posterior probability of H₂ is 4/5.

   The new weights are (1/3)(1/4) = 1/12 and (2/3)(1/2) = 1/3, summing to 5/12. Dividing gives (1/3)/(5/12) = 4/5. Multiplying by the original prior again would count it twice.

4. Use the main prior 2/3, 1/3 but suppose two readouts share exactly the same error E₂ = E₁. Both are 4 m. What is the posterior, and why is squaring the single-reading likelihood unjustified?

   Show the answer and reasoning

   The posterior remains 1/2, 1/2.

   Given either candidate, both equal readouts are determined by one error. The joint event therefore has the single-reading likelihood 1/4 or 1/2. The weights are still 1/6 and 1/6; the joint model does not supply an independent second factor.

5. After the main two conditionally independent 4 m readouts, what is the probability that a fresh readout is 6 m under the same error assumptions? Is it the posterior probability 2/3 of H₂?

   Show the answer and reasoning

   The fresh-reading probability is 1/6.

   H₁ cannot give 6 m under its supplied error law. H₂ can, but requires error +2 m with probability 1/4. The mixture gives (1/3)(0) + (2/3)(1/4) = 1/6. A candidate probability and a readout probability describe different events.

6. Under the exact two-candidate discrete reading model, a stipulated readout is 3 m. What is the normalizer? Can dividing the weights give a posterior of 1/2, 1/2?

   Show the answer and reasoning

   The normalizer is zero, and this Bayes division cannot produce a posterior.

   The required errors would be +1 m under H₁ and −1 m under H₂, each excluded from the supplied error law. Both likelihoods and weights are zero. Dividing by zero is undefined; equal weights of zero do not justify an equal posterior. Check the model and encoding assumptions.

What to carry forward

• An inverse inference depends on a forward model, declared data, parameter restrictions and units.
• Extra assumptions can select among indistinguishable fits, while distinct informative data can add constraints.
• Finite Bayes updating multiplies prior and likelihood weights, then divides by a positive total weight.
• Joint likelihoods factor only with the relevant conditional-independence assumptions.
• Posterior prediction combines the candidate distribution with variation in a fresh readout.
• A coherent computed posterior still requires scrutiny of its model and numerical method for a physical application.