Shannon entropy measures uncertainty in a specified probability distribution.
Key knowledge
For a finite discrete distribution with probabilities pᵢ, Shannon entropy is −Σᵢ pᵢ log₂ pᵢ, measured in bits. A zero-probability term contributes zero by its limiting value.
Values and equations
H(X) = −Σᵢ pᵢ log₂ pᵢ
X
Finite discrete random variable
pᵢ
Probability of outcome i
H(X)
Shannon entropy bits
Conditions and scope
pᵢ≥0 and Σᵢpᵢ=1; logarithm base 2 sets the unit to bits.
This is distributional uncertainty, not semantic meaning, truth or usefulness.
Differential entropy and entropy rates for dependent sequences require separate definitions.
Sources · 7
References
Sources retain their recorded scope. A source link alone does not establish verification of every statement.
Finite-distribution entropy, base-2 units and the zero-probability convention.
2026-10-10 · Compared Shannon's definition with IISc's displayed formula and Stanford's zero-term convention; retained the leading minus sign missing from extracted original-paper text.
Foundational review
2026-10-10 · Fresh primary-source inspection of the complete compact record, including every statement, equation, quantity and material condition, followed by independent AI editorial review. This check does not certify all linked lessons or every passage of every cited publication.
Sources from other places
Scope or convention differs
Finite-distribution Shannon entropy, with zero terms omitted.
The source leaves its logarithm base unstated and its examples match base 10; the repository explicitly defines base 2 and bits.
For a finite discrete distribution with probabilities pᵢ, Shannon entropy is −Σᵢ pᵢ log₂ pᵢ, measured in bits. A zero-probability term contributes zero by its limiting value.
For finite random variables X,Y, I(X;Y)=Σₓ,ᵧ p(x,y)log₂[p(x,y)/(p_X(x)p_Y(y))]=H(X)−H(X|Y). It is nonnegative and zero exactly when X and Y are independent.
For an iid finite-alphabet source, fixed-length block schemes at rates above H(X) bits per symbol can have block error tending to zero. At a fixed rate below H(X), block error tends to one.
For finite X,Y, H(X|Y)=Σy p(y)H(X|Y=y) and H(X,Y)=H(Y)+H(X|Y). Conditioning cannot increase this averaged discrete entropy: H(X|Y)≤H(X), with equality exactly for independence.
For finite distributions p,q, D(p‖q)=Σx p(x)log₂[p(x)/q(x)]≥0, with equality exactly when p=q. It is asymmetric; positive p at a zero q gives infinite divergence.
Von Neumann entropy is −tr rho log2 rho and equals Shannon entropy of the density operator's eigenvalues. For an ensemble {p-x,rho-x}, accessible classical mutual information from a measurement is bounded by chi=S(sum p-x rho-x)−sum p-x S(rho-x). Nonorthogonal states limit distinguishability. Entanglement and conditional quantum entropy produce behavior absent from ordinary classical conditional entropy; one formula cannot replace every quantum resource capacity.
The eigenvalue distribution gives a classical entropy expression.