GLOBAL KNOWLEDGE ARK

Physics · Article

Precision, accuracy, resolution, and significant figures

Compare repeated results, reference differences and display steps, then report calculated values with an explicit rounding convention and appropriate limits.

Previous article: SI units and modern metrology

Back to Physics

After reading, you can

  • State the quantity being measured, its unit and the conditions of comparison.
  • Distinguish agreement among repeated results from closeness to a true or reference value.
  • Calculate a sample mean, an observed range and signed reference differences without calling them an accuracy score.
  • Distinguish a display's resolution from the complete error or uncertainty of a measurement.
  • Count significant figures and preserve their meaning through exact unit conversions.
  • Apply stated elementary reporting rules and round once using nearest with ties to even.
01

Specify what is measured and what is compared

The measurand is the quantity intended to be measured. 'The length of this rod at the specified temperature' is more definite than 'the rod'. A reported value needs a quantity kind and a unit; 7.3 mm is a length value, while 7.3 alone does not identify one. Conditions matter when a quantity or instrument response can change.

Four separate questions guide this lesson: How closely do repeated results agree? How close is a result to the quantity's true value? What changes can the device distinguish? How many digits should be reported? Precision, accuracy, resolution and significant figures address these different questions.

All numbers below are original hypothetical teaching data. The mass comparison uses a stipulated reference r = 100.00 g, treated as exact only for that arithmetic exercise. A real reference has an assigned value and associated uncertainty. Neither repeated agreement nor written digits establishes that reference's truth or the instrument's performance.

02

Precision concerns agreement among repeated results

Measurement precision concerns the agreement among replicate indications or measured values for the same or similar objects under specified conditions. Here a replicate is another measurement of the same mass. Repeatability is precision under a specified set of conditions including the same procedure, operator, measuring system, operating conditions and location over a short time.

Supply two sets of three mass indications. Within each set, all the stated repeatability conditions are held fixed. A gives 100.48, 100.50 and 100.52 g; B gives 99.80, 100.00 and 100.20 g. Each display has a stipulated 0.01 g step. These are observed sample patterns in the model, not a population-level instrument assessment.

The sample mean is the sum of the n observations divided by their exact count n. The observed range is the largest observation minus the smallest. A has mean 100.50 g and range 0.04 g; B has mean 100.00 g and range 0.40 g. A's displayed values cluster more tightly in these samples, despite having the same display step as B.

A range is a simple descriptive spread, not a complete uncertainty estimate or an interval guaranteed to contain a true value. Formal precision studies commonly use a standard deviation or related statistic, with conditions stated. Three readings cannot establish how either system behaves for all masses, times, operators or environments.

03

Accuracy concerns closeness, not the number of digits

Measurement accuracy describes closeness between a measured value and a true value of the measurand. It is a qualitative concept, not a quantity assigned a numerical value such as 'accuracy = 0.02 g'. In practice the true value is generally not known exactly, and comparisons require an appropriate reference and its uncertainty.

For our stipulated reference r, define a signed difference Δ = x − r for an indicated value x. Positive means the indication exceeds the reference; negative means it falls below it. The differences have units g. A's differences are +0.48, +0.50 and +0.52 g; B's are −0.20, 0.00 and +0.20 g.

Every supplied B value is closer to this reference than every supplied A value, yet A has the tighter observed cluster. A mean difference of +0.50 g and B mean difference of 0.00 g are numerical comparisons with the stipulated reference. They are not numeric accuracy values. B's zero mean difference does not erase its spread.

The wider metrological concept of trueness concerns the agreement of an average from indefinitely many replicates with a reference. A three-reading mean alone does not establish trueness, zero bias or a full accuracy claim. Tight clustering can accompany a common offset; adding decimal places cannot remove that offset.

r = 100.00 gx̄_A = 100.50 gx̄_B = 100.00 g
Compare with a referenceMeasurement accuracy describes closeness between a measured value and a true value of the measurand. It is a qualitative concept, not a quantity assigned a numerical value such as 'accuracy = 0.02 g'. In practice the true value is generally not known exactly, and comparisons require an appropriate reference and its uncertainty.

A: 100.48, 100.50, 100.52 g; B: 99.80, 100.00, 100.20 g

A's mass readings have mean 100.50 g and observed range 0.04 g. B's have mean 100.00 g and range 0.40 g. The tighter displayed cluster belongs to A in these samples. With the stipulated exercise reference 100.00 g, A's mean difference is +0.50 g and B's is 0.00 g. These numerical differences are not accuracy scores; B still has a nonzero observed spread.
04

Resolution concerns distinguishable changes

A displaying device's resolution is the smallest difference between displayed indications that can meaningfully be distinguished. General measuring-system resolution concerns a change in the measured quantity that produces a perceptible indication change; noise, friction and the quantity's value can matter. A last displayed digit alone does not establish the complete system's resolving ability.

For a deliberately ideal digital model, let z be an unrounded length indication in millimetres. The display rounds z to the nearest multiple of d = 0.1 mm, using ties to even. Its distinguishable displayed levels are 0.1 mm apart. Both z = 7.26 mm and z = 7.31 mm give the same display, 7.3 mm.

Nearest rounding alone changes z by at most d/2 = 0.05 mm. This is a bound on the numerical rounding operation, not a total measurement-error bound or an evaluated measurement uncertainty. It assumes precisely the supplied ideal display rule.

Now additionally stipulate z = ℓ + b, where ℓ is the model's true length and b = +0.40 mm is a fixed indication offset. For ℓ = 6.90 mm, z = 7.30 mm and the display is 7.3 mm. The displayed difference from ℓ is +0.40 mm, even though the display step is only 0.1 mm and no rounding change occurs in this case.

05

Significant figures describe the written value

Under the elementary reporting convention used here, significant figures are the digits retained in an approximate reported numerical value. Leading zeros locate the decimal point and do not count. Zeros between nonzero digits count, and trailing zeros after the decimal point count. The convention helps communicate reporting detail; it does not certify that particular digits are physically correct.

Scientific notation makes intended trailing zeros explicit. For a positive value, write a × 10ⁿ with 1 ≤ a < 10 and integer n. Count the retained digits in a, not in the exponent. The bare notation 1200 m can leave trailing-zero significance ambiguous; 1.2 × 10³ m, 1.20 × 10³ m and 1.200 × 10³ m explicitly report two, three and four significant figures respectively.

For example, 0.004050 m has four significant figures and its last retained place is 0.000001 m. Using the exact conversion 1 m = 1000 mm gives 4.050 mm, still four significant figures with last retained place 0.001 mm. Replacing it with 4.05 mm discards a reported digit.

An exact count, such as three observations, and a defined unit-conversion factor do not acquire measurement uncertainty from how many digits they have. They therefore do not limit significant figures in the way an approximate measured input can. Distinguish exact data from approximate measured data before applying reporting rules.

Counting digits in stated approximate length reports; equal numerical values can communicate different retained detail.
Reported valueSignificant figuresLast retained place
0.004050 m40.000001 m
4.050 mm40.001 mm
1.20 × 10³ m310 m
1.200 × 10³ m41 m
06

State which reporting rule a calculation uses

When a full uncertainty evaluation is unavailable, elementary exercises often specify approximate significant-figure rules. For addition or subtraction in the same unit, round the final value to the coarsest last retained decimal place among the approximate inputs. For multiplication or division, retain the smallest significant-figure count among those inputs. Exact counts and defined conversion factors do not impose that limit.

These are reporting heuristics, not uncertainty-propagation formulas or guarantees about the output's accuracy. They do not incorporate calibration information, correlations, a measurement model or all uncertainty components. If an evaluated uncertainty and a reporting procedure are supplied, use that information rather than infer an uncertainty from significant figures alone.

For approximate lengths a = 12.4 cm and b = 2.36 cm, the displayed-number sum is 14.76 cm. The addition rule rounds it to 14.8 cm because a's last retained place is 0.1 cm. Their product is 29.264 cm², reported as 29.3 cm² under the multiplication rule because both inputs have three significant figures.

The choice is based on the operation. The sum and product do not use the same criterion, and their quantity kinds differ: the sum is a length and the product is an area. The extra digits in an intermediate calculation describe arithmetic on the supplied numbers, not newly measured information.

07

Round at the end and specify ties

In this lesson, rounding means choosing the nearest value at the requested decimal place. At an exact halfway value, choose the result whose last retained digit is even. Thus 2.345 m to the nearest 0.01 m becomes 2.34 m, while 2.355 m becomes 2.36 m. These exact decimal ties illustrate one documented convention; other reporting procedures may specify another.

Inspect all discarded digits. A value of 2.3451 m is above the halfway value and rounds to 2.35 m, even though its first discarded digit is 5. Truncation simply discards digits and is a different operation.

Retain available intermediate digits and round the final result once. For example, 1.2451 m rounded directly to three significant figures is 1.25 m. First rounding it to four gives 1.245 m; rounding that altered value to three with ties to even gives 1.24 m. The intermediate rounding created a tie and changed the final result.

A displayed result such as 29.3 cm² should retain its unit and its intended digits. Rounding numbers does not establish repeatability, correct an instrument or supply an uncertainty evaluation. Record the original observations so later checking does not depend on rounded summaries.

08

Report the evidence behind the digits

A useful measurement report identifies the measurand, method and relevant conditions, gives the value and unit, and includes an evaluated uncertainty when needed. An uncertainty characterizes the spread of values attributed to the measurand from the information used. It can include effects not visible in repeated readings. Its meaning is not defined by a decimal place or by a ± symbol alone.

When a procedure supplies an appropriately rounded uncertainty, a common reporting step aligns the reported value's last decimal place with that uncertainty, in the same unit. The uncertainty's type and applicable coverage information must remain explicit. NIST's inspected laboratory guidance offers several rounding options; the tie rule chosen here does not prescribe every laboratory's uncertainty-reporting method.

For our examples, the sample range, reference difference, display step and retained digits each provide different information. None alone is the full uncertainty. The next subject, Random and systematic uncertainty, examines sources of uncertainty and how they contribute to a measurement result.

Worked examples

Separate observed spread from reference closeness

Under the stated repeatability conditions, A gives 100.48, 100.50 and 100.52 g, while B gives 99.80, 100.00 and 100.20 g. Use the stipulated exact exercise reference r = 100.00 g to find each sample mean, range and mean reference difference.

  1. There are exactly three observations in each set. A's sum is 301.50 g, so its mean is 301.50 g / 3 = 100.50 g. B's sum is 300.00 g, giving mean 100.00 g. These displayed means are descriptive summaries, not independently evaluated uncertainty reports.
  2. A's observed range is 100.52 g − 100.48 g = 0.04 g. B's is 100.20 g − 99.80 g = 0.40 g. B's range is ten times A's in these supplied samples.
  3. Subtract r from the means: Δ̄_A = +0.50 g and Δ̄_B = 0.00 g. The bar denotes an average. Equivalently, average the three individual signed differences in each set.
  4. Every A difference has magnitude at least 0.48 g, while every B difference has magnitude at most 0.20 g. Thus B's given values are closer to this stipulated reference, while A's given values have a tighter cluster.
  5. Neither the range nor the mean difference is a numerical accuracy score. A reference comparison with real data must account for reference uncertainty, and these three observations do not establish population bias or performance under different conditions.

A: mean 100.50 g, range 0.04 g, mean reference difference +0.50 g. B: mean 100.00 g, range 0.40 g, mean reference difference 0.00 g. Observed repeatability spread and reference closeness differ.

A fine display can retain a larger offset

An ideal length display rounds z to the nearest 0.1 mm with ties to even. First compare z = 7.26 mm and z = 7.31 mm. Then use the supplied model z = ℓ + 0.40 mm with ℓ = 6.90 mm. What does the display step tell us about the complete reference difference?

  1. The nearest display level for both 7.26 mm and 7.31 mm is 7.3 mm. Their rounding changes, display minus z, are respectively +0.04 mm and −0.01 mm. Both lie within the ideal rounding bound 0.05 mm.
  2. In the offset case, z = 6.90 mm + 0.40 mm = 7.30 mm. The display gives 7.3 mm, numerically equal to this unrounded indication, so rounding itself changes nothing.
  3. The displayed difference from the stipulated true length is 7.3 mm − 6.90 mm = +0.40 mm. This exceeds the rounding-only bound because the separate offset was not a rounding error.
  4. More decimal places would describe z more finely but would not remove the stipulated offset. The ideal display model supplies no real calibration assessment or complete uncertainty budget.

The display levels are 0.1 mm apart, but the offset example has a +0.40 mm displayed difference from ℓ. Resolution and rounding alone do not bound total measurement error.

Use decimal places for a sum and significant figures for a product

Take approximate lengths a = 12.4 cm and b = 2.36 cm. Under the stated elementary reporting rules, report a + b and the rectangular area ab. Keep the full displayed-number arithmetic until final rounding.

  1. Both inputs are lengths in the same unit. Their exact displayed-number sum is 12.4 + 2.36 = 14.76, so a + b has units cm.
  2. The coarser last retained decimal place is 0.1 cm, from a. Round 14.76 cm once to that place, giving 14.8 cm. This is the addition convention; counting significant figures is not the sum criterion.
  3. For a rectangle with these supplied side lengths, the displayed-number product is 12.4 × 2.36 = 29.264. Length times length gives cm².
  4. Each approximate input has three significant figures. The elementary multiplication rule therefore reports 29.264 cm² to three significant figures as 29.3 cm².
  5. These reports follow the exercise's conventions. No uncertainties, input correlations or calibration information are supplied, so neither report constitutes an evaluated area or length uncertainty.

The sum is reported as 14.8 cm and the area as 29.3 cm² under the stated rules. The intermediate values 14.76 cm and 29.264 cm² are arithmetic results, not additional measurement precision.

Try it yourself

  1. A third hypothetical mass sample gives 100.00, 100.00 and 100.00 g on a display with 0.01 g steps. Find its mean and observed range. Does it establish zero measurement uncertainty?

    Show the answer and reasoning

    The mean is 100.00 g and the displayed range is 0.00 g. Zero measurement uncertainty is not established.

    All displayed values agree, so their maximum minus minimum is zero. Changes smaller than the display's distinguishable levels and other instrument or reference effects can remain hidden. A zero observed range is not a full uncertainty evaluation.

  2. Relative to the stipulated exercise reference 100.00 g, find the signed differences of 100.48 g and 99.80 g. Which given indication is closer to that reference?

    Show the answer and reasoning

    The differences are +0.48 g and −0.20 g. The 99.80 g indication is closer to the stipulated reference.

    Subtract the reference from each indication, keeping the sign. Closeness compares magnitudes: 0.20 g is less than 0.48 g. These are reference differences, not assigned numerical accuracy values.

  3. In the ideal nearest-0.1 mm display model, let z = 7.34 mm. Find the displayed value and display-minus-z difference. Does the result certify the true length to within 0.05 mm?

    Show the answer and reasoning

    The display is 7.3 mm and its rounding difference is −0.04 mm. It does not certify the true length to within 0.05 mm.

    7.34 is nearer to 7.3 than to 7.4. The numerical rounding difference obeys the model's bound, but z can already contain an offset or other measurement effects. The true length has not been supplied here.

  4. Under the stated reporting convention, count the significant figures in 0.004050 m. Convert it to millimetres using the exact factor and retain all reported significant figures.

    Show the answer and reasoning

    There are four significant figures, and the converted report is 4.050 mm.

    The retained digits are 4, 0, 5 and the trailing 0. Multiplication by the exact factor 1000 mm/m moves the decimal point without adding measured information or limiting significant figures. The last retained place changes from 0.000001 m to the equal length 0.001 mm.

  5. A rectangle has approximate side lengths 5.60 m and 2.1 m. Report the sum of these lengths and their product, the rectangle's area, using the lesson's elementary rules. State both quantity units and make the product's significant figures explicit.

    Show the answer and reasoning

    The sum is 7.7 m. The product is 1.2 × 10¹ m², with two significant figures.

    The displayed-number sum is 7.70 m, rounded to the coarser 0.1 m place. The product is 11.76 m², rounded to two significant figures because 2.1 m has two. Scientific notation explicitly preserves the intended two-digit report. These are reporting conventions, not uncertainty estimates.

  6. Round 1.2451 m directly to three significant figures using nearest with ties to even. Compare with first rounding it to four significant figures and then to three. Why do the results differ?

    Show the answer and reasoning

    Direct rounding gives 1.25 m. The two-stage route gives 1.245 m and then 1.24 m.

    The original value is above the halfway point 1.245 m between 1.24 m and 1.25 m. Intermediate rounding replaces it with that exact tie, where the retained digit 4 is even. Keeping available intermediate digits avoids creating this different final result.

What to carry forward

  • Precision compares replicate results under specified conditions; accuracy concerns closeness to a true value.
  • A sample range and signed reference difference describe different features and are not numeric accuracy scores.
  • A display step and its rounding bound do not determine complete measurement error or uncertainty.
  • Significant figures communicate retained numerical detail; exact counts and defined conversion factors do not impose a measured-digit limit.
  • Elementary addition and multiplication use different reporting criteria, with units retained throughout.
  • Specify the rounding convention, retain intermediate digits and report the measurement context behind the final value.

References

Sources for the concepts and teaching approaches in this article.

  1. International Vocabulary of Metrology (VIM3)Joint Committee for Guides in Metrology (JCGM)Reference guideVIM3, 2.13 Measurement accuracyVIM3, 2.15 Measurement precisionVIM3, 4.15 Resolution of a displaying devicejcgm.bipm.org
  2. GLP 9: Rounding Expanded Uncertainties and Calibration Values (2019)National Institute of Standards and Technology, Office of Weights and MeasuresReference guidenist.govRead source
  3. Physics Part I, Chapter 1: Units and Measurement (Reprint 2026–27)National Council of Educational Research and Training (NCERT), IndiaTextbook chapter · PDFncert.nic.inRead source
  4. What are measurement standards?National Metrology Institute of Japan (NMIJ) — National Institute of Advanced Industrial Science and Technology (AIST)Reference guideunit.aist.go.jpRead source