Floating-point arithmetic and numerical stability

Separate physical values and numerical representations

A physical quantity and the number used to represent it are different things. For a length x = 1.25 m, the numerical value 1.25 refers to metres; the same length is 1250 mm. Record the unit before computing. A program that stores only a number does not automatically retain the intended quantity or unit.

A mathematical model may supply an exact real value, but a finite computer format cannot represent every real number. Distinguish the intended input, the stored value, the computed result and the text displayed to the reader.

The calculations below are original exact teaching cases, not measured lengths or instrument specifications. Their error means a difference from a declared exact mathematical reference. Measurement uncertainty, an approximate physical model and an incorrect unit conversion remain separate questions.

Define a bounded floating-point format

A floating-point format specifies a base, a finite significand and an allowed exponent range. The significand carries the significant digits; the exponent shifts the scale. For example, 7.12 × 10² has significand 7.12 and exponent 2 in base ten. Negative values need a sign, and zero needs its own representation. The precision counts significant digits, not digits after the decimal point.

For hand calculations, define an original decimal teaching format with three significant digits. Its nonzero representable numerical values are ±d.dd × 10ᵉ, where the first digit d is 1–9, the following two digits are 0–9, and the integer exponent e is −2, −1, 0, 1 or 2. Include zero separately. Thus the smallest positive value is 0.0100 and the largest is 999. These are numerical limits in the unit chosen for each calculation.

Define R(q) only for q = 0 or 0.0100 ≤ |q| ≤ 999. It returns the nearest representable value; at an exact midpoint choose the value whose last significand digit is even. Zero returns zero. If a nonzero input or an exact intermediate result is outside that domain, stop and report a range failure. We do not silently replace it with zero or infinity, and we define no subnormal values. This format is a teaching model, not a claim about a particular computer.

For every operation in this model, first calculate its exact numerical result and then apply R. Parentheses specify the order, and each operation is rounded separately. R is our stated mathematical rule, not Python's round function. When a quantity is involved, apply the rule to its numerical value in the declared unit and retain the resulting quantity kind and unit. An undefined exact operation, such as division by zero, supplies no result to round.

Measure numerical error with a reference

Let z be an exact reference result and ẑ its computed approximation, in the same quantity kind and unit. The signed error is ẑ − z, and the absolute error is |ẑ − z|. If z ≠ 0, the relative error magnitude is |ẑ − z|/|z|, a dimensionless ratio. Relative error with a zero reference is undefined; use an appropriate absolute comparison instead. Error analysis below uses exact reference arithmetic; a small error value need not itself be representable in the teaching format.

For an exact supplied length 7.123 m, storing R(7.123) m = 7.12 m gives signed error −0.003 m and absolute error 0.003 m. The relative error magnitude is 0.003/7.123 = 3/7123, about 0.000421, or 0.0421%. The error is calculated from the exact teaching input, not inferred from the number of displayed decimal places.

Spacing changes with scale in this floating format. Between 1.00 and 9.99 the spacing is 0.01; between 100 and 999 it is 1. A small contribution can therefore disappear when added to a much larger value. A local rounding bound for one operation does not automatically bound the error of a complete calculation.

Follow the order of rounded operations

Ordinary real addition is associative: parentheses do not change (a + b) + c = a + (b + c). Rounded arithmetic can give different answers because each intermediate result is mapped back into the format. Track every operation rather than rounding only the final exact sum.

Supply signed position changes along one fixed axis with numerical metre values a = 100, b = 0.200 and c = −100. Their exact real sum is 0.200 m. For the left grouping, R(100 + 0.200) = 100, then R(100 − 100) = 0. For the right grouping, R(0.200 − 100) = −99.8, then R(100 − 99.8) = 0.200. Every listed nonzero result is within our declared range.

The right grouping preserves the small contribution in this case. This does not prove that one grouping is always best or that arbitrarily reordering a real program is harmless. The actual format, operations, compiler and software implementation need to be stated and checked; our hand calculation specifies its rounding path explicitly.

Inspect cancellation and rearrange carefully

Subtracting nearly equal values cancels their leading digits. If those values already contain approximation errors, their difference can have a much larger relative error. The subtraction itself need not introduce an error: it can expose earlier rounding.

For an original area comparison, supply exact square side lengths a = 1.01 m and b = 1.00 m. The exact difference of their areas is D = a² − b² = 0.0201 m². In our format, round the numerical results in m²: R(1.01 × 1.01) = 1.02 and R(1.00 × 1.00) = 1.00. Their rounded difference is R(1.02 − 1.00) = 0.0200. The final subtraction is exact for those rounded operands, but the area error is −0.0001 m².

The identity a² − b² = (a − b)(a + b) suggests another path. The numerical metre differences and sums are R(1.01 − 1.00) = 0.0100 and R(1.01 + 1.00) = 2.01. Their product in m² gives R(0.0100 × 2.01) = 0.0201, the exact reference for this case. Both the units and every intermediate rounding matter.

Use this as a checked improvement for these exact inputs, not a theorem that the factored path is exact or stable for all inputs. A rearrangement cannot recover information already lost in uncertain or rounded input data, and it can introduce other range or domain problems.

Distinguish conditioning and stability

Conditioning concerns the mathematical problem: how much its exact output changes when its specified inputs change. It depends on the input values and on how changes are measured. A problem may be sensitive even if every arithmetic operation is performed exactly. Higham describes this distinction through condition numbers, which compare input and output sensitivity.

For the exact length-difference problem d = a − b, take a = 1.04 m and b = 1.03 m, so d = 0.0100 m. Change the supplied inputs to a′ = 1.045 m and b′ = 1.025 m. Each input changes by 0.005 m, less than 0.5% of its original magnitude, but d′ = 0.0200 m: a 100% change relative to the original difference. This is a finite sensitivity calculation in exact real arithmetic, separate from our rounded-operation examples.

Stability concerns the algorithm: how errors introduced during its execution affect the result. One important definition, backward stability, asks whether the computed result is the exact solution for slightly perturbed input data, with smallness assessed against the arithmetic precision and stated error measure. A sensitive problem can have a stable algorithm while still producing a relatively sensitive output.

The original sum and area cases compare specified calculation paths; they establish no general backward-stability bound. Conversely, changing to a higher-precision format can reduce representation and operation errors without reducing uncertainty in the supplied physical data. Keep the exact problem, its input sensitivity and the implemented algorithm distinct.

Inspect binary floats and declare comparisons

A finite binary fraction is an integer divided by a power of two. The exact decimal number 0.125 = 1/8 therefore has a finite binary representation; 0.1 = 1/10 does not, because its reduced denominator includes a factor of five. The Python tutorial explains this representation issue and distinguishes the stored binary value from its shortened decimal display. This does not make every stored value inexact.

On the checked CPython interpreter with base-two, 53-bit float precision, the expression 0.1 + 0.2 displays 0.30000000000000004. This is a checked example for the stated float environment; our three-digit decimal format is a separate model.

The comparison (0.1 + 0.2) == 0.3 evaluates to False in that environment. Equality tests the stored values, not the equality of their intended decimal real inputs.

For finite comparable values a and b, Python's math.isclose uses the condition |a − b| ≤ max(r max(|a|, |b|), A), where r is the relative tolerance and A the absolute tolerance. Use nonnegative tolerances with r < 1 here. If the values represent quantities, both numerical inputs and A must refer to the same chosen unit, while r is dimensionless. The function does not attach units for you.

A purely relative tolerance cannot accept a nonzero value against zero when r < 1. A suitable positive absolute tolerance can. Choose tolerances for the stated numerical test and its purpose; the defaults or an accepted comparison do not establish measurement uncertainty, model validity or a stability proof. Nonfinite values require separate checks.

Check the calculation and report its limits

Before interpreting a numerical result, retain its input meanings and units, the number format and range, the operation order and the comparison rule. Compare against an exact small case when one is available. Record range failures and nonfinite results rather than treating every returned number as a physical answer.

Changing precision or using an algebraically equivalent path can reveal a numerical problem. Agreement between two outputs is useful evidence to inspect, but it does not prove their common physical assumptions or rule out a shared mistake. Document which inputs were exact stipulations and which came from measurement, and avoid reporting numerical digits as a physical uncertainty estimate.

This lesson has checked finite-format behavior, one useful rearrangement and an exact sensitivity example. The next map subject, Root finding and optimization, uses numerical procedures to find solutions and extrema; representation, algorithm conditions and stopping criteria remain part of the evidence needed to interpret their outputs.

Worked examples

Try it yourself

1. Find R(8.765) and R(8.775) in the decimal teaching format. What should happen to exact inputs 0.001 and 1100?

   Show the answer and reasoning

   R(8.765) = 8.76 and R(8.775) = 8.78. The other two inputs cause declared range failures.

   Each first value is halfway between adjacent hundredths; select the even last significand digit, 6 or 8. The nonzero 0.001 is below 0.0100 and 1100 exceeds 999. Our rule does not silently map them to zero or infinity.

2. An exact supplied length 6.283 m is represented as 6.28 m. Find signed, absolute and relative error. Can this relative-error formula be used with an exact zero reference?

   Show the answer and reasoning

   The signed error is −0.003 m, absolute error 0.003 m and relative error magnitude 3/6283, about 0.0477%. The zero-reference relative error is undefined.

   Subtract the exact reference from the approximation, then divide the error magnitude by 6.283 m. The units cancel in the ratio. Dividing by zero is not an error measure; state an absolute comparison instead.

3. Sum signed position changes along one fixed axis with numerical metre values 100, 0.400 and −100 using (a + b) + c and a + (b + c), applying R after every operation. Compare with the exact sum.

   Show the answer and reasoning

   The first grouping gives 0 m; the second gives 0.400 m, equal to the exact reference.

   R(100.400) = 100, then R(100 − 100) = 0. In the other grouping, R(0.400 − 100) = −99.6, then R(100 − 99.6) = 0.400. All intermediates meet the format's domain.

4. For exact square side lengths 2.01 m and 2.00 m, compute the area difference directly and as (a − b)(a + b), rounding every operation in the stated units.

   Show the answer and reasoning

   Direct evaluation gives 0.0400 m²; the factored path gives the exact 0.0401 m².

   The exact squares are 4.0401 and 4.0000 m², rounded to 4.04 and 4.00; their difference is 0.0400 m². The factored path uses the exact represented 0.0100 m and 4.01 m, whose product 0.0401 m² is representable. This is another checked case, not a proof over all inputs.

5. For the exact problem d = a − b, change a = 1.04 m and b = 1.03 m to a′ = 1.035 m and b′ = 1.035 m. What is the output change? Does it require rounding error?

   Show the answer and reasoning

   The difference changes from 0.0100 m to 0 m, an absolute change of 0.0100 m and a 100% change relative to the original difference. No rounding error is required.

   The inputs change by −0.005 m and +0.005 m, while exact subtraction changes the small difference by its full original magnitude. This is sensitivity of the exact problem to its supplied inputs; computing with more digits does not undo that input change.

6. After importing math, compare the finite value 10⁻¹⁰ with zero using math.isclose(1e-10, 0.0, rel_tol=1e-9, abs_tol=0.0). Then repeat with relative tolerance zero and absolute tolerance 10⁻⁹. Explain the outcomes and their scope.

   Show the answer and reasoning

   The relative-only call returns False; the absolute-tolerance call returns True. Neither is a statement of measurement uncertainty.

   For the first call, a nonzero difference cannot satisfy |a| ≤ r|a| with r < 1. The second call is math.isclose(1e-10, 0.0, rel_tol=0.0, abs_tol=1e-9): it allows a difference below 10⁻⁹ in the chosen numerical unit. These tolerances define the supplied check; they neither validate a physical zero nor prove algorithmic stability.

What to carry forward

• Specify quantity meanings, units, format precision, range and rounding before interpreting a computed number.
• Finite representation and repeated rounding can change results even when every individual operation follows its specified rule.
• Use a nonzero exact reference for relative numerical error; near zero, state an appropriate absolute comparison.
• Cancellation can expose errors already present in operands, while a checked rearrangement may improve a particular calculation.
• Conditioning describes input sensitivity of the exact problem; stability describes the algorithm's handling of execution errors.
• Keep code and environment records literal, declare comparison tolerances and distinguish numerical checks from physical uncertainty and model validation.