Shannon entropy of a discrete distribution
Shannon entropy measures uncertainty in a specified probability distribution.
Key knowledge
For a finite discrete distribution with probabilities pᵢ, Shannon entropy is −Σᵢ pᵢ log₂ pᵢ, measured in bits. A zero-probability term contributes zero by its limiting value.
Values and equations
H(X) = −Σᵢ pᵢ log₂ pᵢ
- X
- Finite discrete random variable
- pᵢ
- Probability of outcome i
- H(X)
- Shannon entropy bits
Conditions and scope
- pᵢ≥0 and Σᵢpᵢ=1; logarithm base 2 sets the unit to bits.
- This is distributional uncertainty, not semantic meaning, truth or usefulness.
- Differential entropy and entropy rates for dependent sequences require separate definitions.
Sources · 7
References
Sources retain their recorded scope. A source link alone does not establish verification of every statement.
- A Mathematical Theory of Communication (1948), §6: Choice, uncertainty and entropyClaude E. Shannon, Bell Telephone Laboratories; reprint hosted by Yale UniversityResearch paper · PDF
- MA261 notes, chapter 27, Definition 145: EntropySiddhartha Gadgil / Indian Institute of ScienceCourse notes
John Duchi, Stanford University2 sources
- A Mathematical Theory of Communication (1948)Claude E. Shannon / Bell System Technical JournalResearch paper · PDF
- Statistics 311 lecture notes, entropy and relative entropyStanford UniversityCourse notes · PDF
- Entropy, Diversity and MagnitudeEmily Roff, University of EdinburghTeaching resource · PDF
Evidence and review
Finite-distribution entropy, base-2 units and the zero-probability convention.
2026-10-10 · Compared Shannon's definition with IISc's displayed formula and Stanford's zero-term convention; retained the leading minus sign missing from extracted original-paper text.
Foundational review
2026-10-10 · Fresh primary-source inspection of the complete compact record, including every statement, equation, quantity and material condition, followed by independent AI editorial review. This check does not certify all linked lessons or every passage of every cited publication.
Sources from other places
Scope or convention differs
Finite-distribution Shannon entropy, with zero terms omitted.
The source leaves its logarithm base unstated and its examples match base 10; the repository explicitly defines base 2 and bits.
University of Edinburgh, School of Mathematics
- Entropy, Diversity and Magnitude United KingdomEnglish
Entropy, PDF pp.8–13
- Entropy, Diversity and Magnitude
Entry details
Permanent link to this entry ↗- Record identity
- info-shannon-entropy
- Revision
- core.1
- Structured reference record
- Essential core · first collection