Probability and statistical entropy

Two spins, one count, two preparations

Two localized spins are measured along a fixed axis. Each gives down (D) or up (U). The sites are distinguishable, so DU and UD are different microscopic outcomes. In preparation A each spin has up probability p = 1/2; in preparation B it has p = 4/5. Within each preparation the two outcomes are independent. These are assigned preparation probabilities, without assuming a thermal reservoir or an equilibrium law.

A detector reports only M, the number of up spins. Which is more likely: one up spin or two? Does counting the configurations settle the question? There are two configurations with M = 1 and only one with M = 2, but their individual weights matter. The table lists every microscopic outcome; every column sums to one.

For A, M = 1 is most probable. For B, M = 2 is most probable even though it has smaller multiplicity. The aim is to understand this change and give a precise meaning to the entropy of a probability distribution.

Choose a reading path

On a first reading, follow the two-spin probabilities, the definition of entropy, the difference between microstate and macrostate entropy, and the two coarse-graining operations. Use their tables before the algebra. Then compare two reference distributions using relative entropy. Sections headed “Further” supply proofs and continuous-variable cautions; return to them after the finite examples.

The published preparation “Algebra, functions, graphs, and logarithms” explains the logarithms and function notation used here. “Physical systems, states, boundaries, and environments” helps identify what the system includes. The optional advanced path uses “Integrals and multivariable calculus” and “Hamiltonian mechanics and phase space”. The preparation links above lead to those existing English lessons.

Here a microstate is one outcome at the chosen microscopic resolution; a macrostate is a set of microstates sharing a specified coarse description. Probability is a normalized assignment to mutually exclusive outcomes. An ensemble is the distribution describing a repeated preparation or a statistical model; it does not require many physical copies to be present at once. A single observed outcome is distinct from the distribution used to predict it.

Sum events and condition on information

For a finite, exhaustive list, p_i ≥ 0 and Σ_i p_i = 1. An event is a subset of that list. Sum p_i over the subset to obtain its probability. Add probabilities directly only for disjoint events; in general P(A ∪ B) = P(A) + P(B) − P(A ∩ B).

For events with P(B) > 0, P(A|B) = P(A ∩ B)/P(B). Conditioning changes the question. Given M = 1 in either opening preparation, DU and UD each have conditional probability 1/2. This follows from their equal assigned weights, not merely their different names. No conditional probability on a zero-probability event is defined by this quotient.

Independence means P(X,Y) = P(X)P(Y), not just that the sites are separated. The opening preparation explicitly grants it, giving P(DD) = (1−p)² and P(DU) = P(UD) = p(1−p). For N independent identical two-outcome sites, P(M = m) = C(N,m)p^m(1−p)^(N−m), where C(N,m) = N!/[m!(N−m)!] and m = 0,…,N. Its mean is Np and its variance is Np(1−p). Correlations or nonidentical site probabilities require a different distribution.

State the weighting before counting

Equal a priori probability is a physical modeling assumption for accessible microstates under specified constraints and a specified measure. For W equally weighted discrete states, p_i = 1/W and an event containing Ω of them has probability Ω/W. Equal counts of arbitrary coordinate bins need not have equal physical weight.

The opening states have equal weights only in preparation A. In B, UU has probability 0.64 while DD has 0.04. For a system exchanging energy with a reservoir, different energy states generally carry different equilibrium weights. The later ensemble lesson derives those weights. Neither the word “random” nor lack of a measured trajectory establishes uniformity, independence or dynamical exploration of every accessible state.

In classical phase space, specify the volume measure and the conserved constraints before assigning weights. Equal phase-space volumes are the relevant equal cells under that model, not equal widths in every possible coordinate. For indistinguishable particles, arbitrary particle labels must not create spurious extra physical states. The preceding multiplicity lesson develops state counting; the present two-site example avoids exchanging identical particles by using distinguishable localized sites.

Average surprisal defines entropy

For an outcome with p_i > 0, its surprisal is I_i = −ln p_i. A rare outcome has larger surprisal than a common one. Shannon entropy is the probability-weighted average H = Σ_i p_i I_i = −Σ_i p_i ln p_i. H describes the whole distribution, not the surprise of the one outcome that happened.

We use natural logarithms: H is a dimensionless number reported in nats. For bits, divide by ln 2: H_bits = H/ln 2. A definite distribution has H = 0. Use the limit 0 ln 0 = 0 for zero-probability terms; do not ask a calculator for ln 0 or replace zero probabilities by arbitrary small numbers.

For one spin, h(p) = −p ln p − (1−p)ln(1−p). Its endpoint values are zero, and h(p) = h(1−p). The maximum is ln 2 at p = 1/2. Bias lowers the average uncertainty even though the unlikely outcome is individually more surprising. The curve and samples refer to one spin, whereas the opening microscopic entropy describes two.

Connect statistical and physical entropy

For a classical distribution over specified physical microstates, Gibbs entropy is S_G = k_B H = −k_B Σ_i p_i ln p_i. Boltzmann’s constant k_B = 1.380649 × 10⁻²³ J/K is exact in SI, so S_G has units J/K. The two-spin assigned model gives the values below. These are model Gibbs entropies, not calorimetric measurements of an unspecified preparation apparatus.

If a distribution is uniform over W accessible physical microstates, S_G = k_B ln W. Boltzmann macrostate entropy instead assigns S_B(M) = k_B ln Ω(M) to the number or measure of microstates compatible with a particular macrostate, with the cell/measure convention fixed. A macrostate can have large S_B even when its label is known with certainty.

For uniform probabilities within each macrostate, S_G = k_B H(M) + Σ_M P(M)S_B(M). If conditional microstate weights are nonuniform, replace ln Ω(M) by their actual conditional entropy. Therefore −k_B Σ_M P(M)ln P(M) of the detector’s labels is generally not the whole thermodynamic entropy. Thermodynamic identification also requires the appropriate physical state, constraints and degrees of freedom; multiplying an arbitrary message entropy by k_B does not establish that identification.

For the opening spins we treat diagonal, classical outcome probabilities only. In a general quantum system, measurement probabilities depend on the measurement basis; their Shannon entropy need not equal the state’s von Neumann entropy. Quantum states and indistinguishability require the later quantum-statistical treatment.

Further: bounds, concavity and constraints

For a fixed finite W-state space, 0 ≤ H ≤ ln W. The lower bound follows term by term because ln p_i ≤ 0. To see the upper bound, compare p with the uniform reference u_i = 1/W: D(p||u) = ln W − H ≥ 0, using the relative-entropy proof below. Equality at the upper bound requires p = u. Changing W or the state resolution changes the comparison.

For the binary case, h′(p) = ln[(1−p)/p] and h″(p) = −1/[p(1−p)] < 0 for 0 < p < 1. Thus the interior stationary point p = 1/2 is the unique maximum, while the derivative is not finite at the endpoints. More generally −x ln x is concave, so mixing normalized distributions cannot lower entropy below their probability-weighted average. Strictness can fail when the distributions agree; “every averaging strictly increases entropy” is too strong.

Maximum entropy is an inference with explicit constraints and a reference measure, not a mechanism forcing a system to relax. For positive reference weights r_i, maximize −Σ_i p_i ln(p_i/r_i) with normalization and specified averages Σ_i p_i f_ai = F_a. At an interior solution, differentiation gives p_i ∝ r_i exp(−Σ_a λ_a f_ai); multipliers enforce the constraints and make every exponent dimensionless. Zero weights restrict the support. Boundary constraints, unbounded continuous domains or incompatible averages may prevent an interior or normalizable solution. The ensemble lesson supplies physical energy and particle-exchange constraints rather than guessing them here.

Correlations change additivity

A joint distribution p_ij describes two outcomes X and Y together. Marginals are p_i = Σ_j p_ij and p_j = Σ_i p_ij. The conditional entropy H(Y|X) averages the entropy of p(j|i) with weight p_i. Expanding ln p_ij = ln p_i + ln p(j|i) gives the chain rule H(X,Y) = H(X) + H(Y|X). Zero-weight conditions contribute zero.

For independent variables, H(Y|X) = H(Y), so entropies add. Correlations reduce joint entropy below the sum of marginal entropies. Their mutual information I(X;Y) = H(X)+H(Y)−H(X,Y) = D(p_XY||p_X p_Y) is nonnegative and zero exactly at independence. This is classical information, not an assertion about energy exchange or a thermodynamic production rate.

Compare two fair independent spins with a classical preparation that gives DD or UU with equal probability and never DU or UD. Each individual spin is still fair in the correlated preparation, but knowing one reveals the other. Marginal measurements alone do not determine joint entropy or justify an independence approximation.

Merge labels without inventing new states

Suppose four distinguishable physical cells have probabilities p = (0.10, 0.20, 0.30, 0.40). A detector reports only A = {1,2} or B = {3,4}. The reported probabilities are P_A = 0.30 and P_B = 0.70. This deterministic many-to-one map replaces four outcome labels by two.

The chain rule gives H(p) = H(P) + Σ_a P_a H(p_i|a). The within-group term is nonnegative, so the entropy of the reported group label is no larger than that of the original microscopic label. Here H(P) = 0.610864 nats while H(p) = 1.279854 nats. Information about which member occurred has been omitted; this statement does not claim the physical system’s thermodynamic entropy fell.

To predict later measurements that depend on the omitted cell, the group probabilities alone are insufficient. One needs the conditional distribution or an explicit reconstruction assumption. Coarse observation and assumed redistribution on the original space are different steps.

Reconstruct on the same microscopic space

Now retain the original four-cell space but replace the unknown conditional distribution inside each group by a uniform one. The reconstruction is q = (0.15,0.15,0.35,0.35). This uses equal physical reference weight for the two cells in each group. For unequal cell measures, uniform density means probabilities proportional to those measures, not equal numerical cell probabilities.

With g_a equally weighted cells in group a, q_i = P_a/g_a. On this same microscopic space, H(q) = H(P) + Σ_a P_a ln g_a. Subtracting H(p) gives H(q)−H(p) = D(p||q) ≥ 0. Reconstruction erases internal information and raises, or leaves unchanged, the microscopic entropy. It is a different comparison from merging labels, which lowers the entropy of the observed label.

Here H(q) = 1.304011 nats, only 0.024157 nats above H(p). The within-group information omitted by merely reporting A or B is 0.668990 nats. These two differences are not interchangeable. The table separates the spaces and the operations so the vertical bars in an entropy comparison cannot be mistaken for a time trajectory.

Relative entropy keeps the reference visible

For normalized p and q on the same discrete space, define the Kullback–Leibler relative entropy D(p||q) = Σ_i p_i ln(p_i/q_i). The first argument supplies the average; the second is the reference model. A zero p_i contributes zero. If p_i > 0 where q_i = 0, D is +∞: the reference excludes an outcome that the preparation allows. This is a support mismatch, not a large finite plotted value.

For positive q on the support of p, −ln x ≥ 1−x gives D(p||q) ≥ Σ_(p_i>0)(p_i−q_i) ≥ 0. Equality requires p = q. D is generally asymmetric and does not satisfy the triangle inequality, so it is not a metric distance. Logarithm base changes its units, not the ordering of its arguments.

For a binary preparation with up probability p and a fixed binary reference with up probability q, D = p ln(p/q)+(1−p)ln[(1−p)/(1−q)]. The two curves use q = 1/2 and q = 4/5. They are zero at their own reference distributions. A biased preparation need not be “further from equilibrium”: equilibrium has not been specified, and the appropriate reference may itself be biased.

Only for the uniform reference on W states is D(p||u) = ln W−H(p). A nonuniform reference requires the full probability ratios. Do not identify the historical phrase “relative entropy” used for a normalized entropy ratio in some early information literature with this KL definition.

Further: probability maps and irreversibility

A stochastic map has entries K_ji ≥ 0 with Σ_j K_ji = 1 for each input i. It transforms both p and q by the same rule. The log-sum inequality gives D(Kp||Kq) ≤ D(p||q). For one output j, apply Σ_i a_i ln(a_i/b_i) ≥ (Σ_i a_i)ln[(Σ_i a_i)/(Σ_i b_i)] to a_i = K_ji p_i and b_i = K_ji q_i; then sum over j. The log-sum inequality itself follows by setting A = Σ_i a_i, B = Σ_i b_i and subtracting its right side: for positive totals the difference is A D(a_i/A || b_i/B), which is nonnegative by the preceding proof. Zero terms use the support conventions above. This proves that the same probabilistic processing cannot improve distinguishability from the reference.

If q is stationary, Kq = q, the result becomes D(Kp||q) ≤ D(p||q). Ordinary H increases under a doubly stochastic map on the same finite state space because the uniform u is then stationary. An arbitrary stochastic map need not preserve u. For example, a reset map that outputs up with probability 4/5 regardless of its input takes a fair spin to that biased distribution, reducing H from 0.693147 to 0.500402 nats. It is not an isolated reversible Hamiltonian evolution.

For a closed classical Hamiltonian system, Liouville evolution preserves phase-space volume and transports the fine-grained density without changing its value along trajectories. With a fixed dimensionless phase-space reference measure and suitable boundary/integrability conditions, −k_B ∫ρ ln ρ dμ is conserved. Macroscopic entropy increase therefore needs a specified coarse description and physical assumptions about preparation, relaxation, correlations or environment. It does not follow from counting alone, from an observer becoming uncertain, or from a claim that every possible coarse-grained time step is monotone. Small systems also permit fluctuations.

For an open system relaxing toward a thermal reference, relative entropy can connect to an appropriate free-energy difference when the reservoir, energies and equilibrium reference are specified. That thermodynamic identification and the distinction between system and total entropy belong to the later ensembles and nonequilibrium lessons. The probability contraction proved here, by itself, is not a complete entropy-production formula.

Further: continuous probabilities need a measure

For a continuous variable x, probability of an interval is ∫f(x)dx; a density is not a point probability and can exceed one. Its units are inverse units of x. To avoid taking the logarithm of a dimensional density without a convention, first specify a dimensionless coordinate or a physical reference measure. A raw differential entropy h = −∫f ln f is not an absolute state count.

Let y = x/(1 m) be dimensionless and uniform on 0 ≤ y ≤ 1/2, with density f_y = 2. Then h_y = −ln 2 = −0.693147. With z = 100y the support is [0,50], f_z = 0.02 and h_z = ln 50 = 3.912023. The probabilities have not changed, yet h_z−h_y = ln 100. In general an invertible smooth coordinate change gives h_z = h_y + E[ln|dz/dy|], when these quantities exist.

Discrete histogram entropy also depends on resolution. Here equal aligned bins of width Δy = 1/4 give two equal occupied bins and H = ln 2; Δy = 1/8 gives four and H = ln 4. For sufficiently regular densities and small equal bins, H_bins ≈ h_y−ln Δy. This relation is exact in this uniform aligned example, not for arbitrary irregular densities or bins.

Continuous relative entropy D(f||g) = ∫f ln(f/g)dy uses densities with respect to the same measure. A finite value requires f = 0 almost everywhere on the set where g = 0, and f ln(f/g) must be integrable. Changing densities on a set of zero integration measure does not affect these integrals. For this f on [0,1/2] and reference g = 1 on [0,1], D = ln 2. Transforming both densities consistently to z cancels the Jacobians, so D remains unchanged. The physical phase-space cell/reference convention, including quantum state counting when needed, is what permits a statistical entropy to be compared with thermodynamics.

Use probability with measurements and models

In repeated spin readout, frequencies estimate the probabilities of the chosen measurement outcomes. Finite data are not the probability law itself: an unobserved outcome can still have nonzero probability. Report the preparation, detection error model, number of samples, time dependence and possible correlations before interpreting an estimated entropy. A finite-record zero is not automatically a physically forbidden state.

In a simulation, a histogram is tied to the bins and sampling procedure. Compare like resolution and like physical measures, check normalization, and account for correlations between samples. Entropy of a few observed categories is not automatically the thermodynamic entropy of all unresolved molecular states. In statistical inference, KL can compare a candidate model with a stated reference; a finite, nonnegative value does not prove either model is physically correct.

Use this sequence: name the physical outcomes and reference measure; justify or estimate their weights; test normalization and independence; say whether entropy concerns a microstate distribution, a macrostate label or a reconstruction; then state the logarithm base and units. Before inferring evolution or equilibrium, supply the dynamics and constraints that support that inference.

Continue with the physical ensemble

“Microstates, macrostates, and multiplicity” develops the state-counting side. “Microcanonical, canonical, and grand canonical ensembles” derives equilibrium probability laws from constraints and exchanges. “Partition functions” makes those laws useful for thermodynamic calculations. “Fluctuations and response functions” and “Non-equilibrium and stochastic thermodynamics” connect probabilities to measurements and dynamics. Follow their permanent map links until their actual English articles are available.

The central result here is usable before those developments: the same list of accessible outcomes can support different probability distributions and different entropies. Counts, weights, correlations, measure and resolution all have physical meaning. Keeping them explicit prevents a mathematically correct entropy calculation from answering the wrong physical question.

Worked examples

Try it yourself

1. A proposed four-outcome model assigns (0.1,0.2,0.3,0.5). Can you compute its probability entropy as given?

   Show the answer and reasoning

   It is not a normalized distribution: the sum is 1.1.

   Find the missing assumption or correct the model first. Dividing by 1.1 would define a different model and needs justification; do not silently repair measured or theoretical weights.

2. In the p = 4/5 opening model, find P(first spin up | M = 1).

   Show the answer and reasoning

   1/2.

   The conditioning event contains DU and UD, each with probability 4/25. Only UD has the first spin up, so the ratio is (4/25)/(8/25) = 1/2.

3. Do DD and UU have equal probabilities merely because each has multiplicity one?

   Show the answer and reasoning

   No; in preparation B their probabilities are 0.04 and 0.64.

   A count is not a probability law without equal weighting. The preparation fixes a bias that gives different product weights.

4. What are h(0), h(1) and h(1/2), in nats?

   Show the answer and reasoning

   0, 0 and ln 2 ≈ 0.693147.

   Use the limit 0 ln 0 = 0 at a definite outcome. At p = 1/2 the two equally likely outcomes maximize binary entropy.

5. Two fair independent spins have two bits of microscopic entropy. What is S_G?

   Show the answer and reasoning

   S_G = 2 k_B ln 2 ≈ 1.913985923 × 10⁻²³ J/K.

   Convert bits to natural-log units before multiplying by k_B. Using 2k_B alone mixes logarithm conventions.

6. Two fair spin marginals are observed. Is joint entropy necessarily 2 ln 2?

   Show the answer and reasoning

   No. The DD/UU correlated preparation has joint entropy ln 2.

   The sum of marginal entropies is an upper bound. Equality requires independence, which marginal measurements alone do not prove.

7. If the four-cell distribution already equals (0.15,0.15,0.35,0.35), does uniform reconstruction within A={1,2}, B={3,4} strictly raise its entropy?

   Show the answer and reasoning

   No; it leaves the distribution and entropy unchanged.

   It is already uniform within both groups, so p = q and D(p||q) = 0. Entropy increase is nonnegative, not necessarily strict.

8. For p=(1/2,1/2) and q=(1,0), is D(p||q) finite?

   Show the answer and reasoning

   No. It is +∞.

   The second outcome is allowed by p but forbidden by q. Replacing q=0 with a small cutoff changes the reference and must not be presented as the exact model.

9. Compare D(p||q) and D(q||p) for p_up = 1/2, q_up = 4/5.

   Show the answer and reasoning

   D(p||q) ≈ 0.223144; D(q||p) ≈ 0.192745 nats.

   The averaging distribution changes when the arguments are swapped. Relative entropy is not a symmetric distance.

10. A reset map outputs up with probability 4/5 for every input. Does it raise the entropy of a fair input spin?

    Show the answer and reasoning

    No. It lowers H from ln 2 to h(4/5), a decrease of about 0.192745 nats.

    The map is stochastic but not doubly stochastic. It does not preserve the uniform distribution; physical implementation and environmental entropy require additional modeling.

11. For the uniform y on [0,1/2], halve the equal bin width from 1/4 to 1/8. What happens to histogram entropy?

    Show the answer and reasoning

    It rises from ln 2 to ln 4, an increase of ln 2.

    The description now distinguishes four equal-probability cells instead of two. This changes the observation resolution without establishing a physical entropy-production process.

12. A closed Hamiltonian ensemble stretches into fine phase-space filaments. Does this by itself increase its fine-grained Gibbs entropy?

    Show the answer and reasoning

    No, under volume-preserving Hamiltonian evolution with the stated regularity and boundary conditions it is conserved.

    A fixed-resolution reconstruction can omit filament information and have a different entropy. Its growth requires its own coarse-graining and physical assumptions; it is not a contradiction of Liouville’s theorem.

13. Can a probability density equal 2 while total probability remains one?

    Show the answer and reasoning

    Yes: f_y=2 on an interval of dimensionless length 1/2 integrates to one.

    Densities are not point probabilities. Their magnitude and units depend on the coordinate; interval probabilities, with the integration measure, are the normalized quantities.

14. A preparation is closer in KL to q_up=4/5 than to q_up=1/2. Does that prove it is closer to thermal equilibrium?

    Show the answer and reasoning

    No.

    A reference is an explicit probability model. To call it thermal equilibrium requires the physical Hamiltonian, constraints and reservoir conditions, which are not supplied by that numerical comparison.

What to carry forward

• Specify the physical state space, measure and probability weights before using multiplicity as a probability.
• H = −Σ p ln p averages surprisal; natural-log Gibbs entropy is S_G = k_B H for the specified physical microstate model.
• Entropy adds for independent variables; correlations require the joint distribution and conditional entropy.
• Merging outcome labels and reconstructing on the original microscopic space are different coarse-graining operations with different entropy comparisons.
• KL relative entropy keeps the reference explicit, is nonnegative and generally asymmetric, and is infinite for a positive-p/zero-q support mismatch.
• Resolution, coordinates, constraints and dynamics determine which statistical entropy can answer a physical thermodynamic question.