GLOBAL KNOWLEDGE ARK

Reference library

Foundations

Explore concepts, laws, equations and methods, with clear explanations of their use and sources. Choose an area of study or find a specific entry.

More filters

Reference entries

Showing 31–60 of 1709 reference entries

Physics · MeasurementCaesium-133 defining frequency

ΔνCs denotes the unperturbed ground-state hyperfine transition frequency of caesium-133. Its numerical value is fixed exactly in SI units.

DefinitionMathematics & logic · InformationClassical propositions and connectives

Classical logic combines truth-valued propositions using defined connectives.

DefinitionMathematics & logicAxioms, deduction and mathematical proof

A mathematical proof derives a conclusion within stated assumptions and rules.

MethodMathematics & logic · InformationSets, membership and basic operations

A set specifies membership without counting duplicate listings.

DefinitionMathematics & logic · InformationFunctions, domain and codomain

A function assigns exactly one output to each input in its domain.

DefinitionMathematics & logic · InformationMathematical induction

A base case and general successor step establish a predicate on all nonnegative integers.

MethodMathematics & logic · MeasurementEuclidean right triangles and the Pythagorean theorem

In a Euclidean right triangle, squared leg lengths sum to the squared hypotenuse length.

Mathematical statementMathematics & logic · PhysicsSine, cosine and the unit circle

Sine and cosine are unit-circle coordinates whose squared values sum to one.

Mathematical statementMathematics & logic · InformationComplex numbers

A complex number has real coordinates a and b and an imaginary unit satisfying i²=−1.

DefinitionMathematics & logic · Information · PhysicsLinear maps and matrix representation

A linear map preserves linear combinations and has a matrix once bases are chosen.

DefinitionMathematics & logicLimit of a real function

A finite limit controls output error through sufficiently small input distance.

DefinitionMathematics & logic · PhysicsDerivative at a point

A derivative is a finite limit of local difference quotients.

DefinitionMathematics & logic · PhysicsDefinite integral and signed accumulation

A definite integral is the limiting signed accumulation of small contributions.

DefinitionMathematics & logic · PhysicsFundamental theorem of calculus

Continuity connects differentiation with integration.

Mathematical statementMathematics & logic · InformationProbability space and axioms

A probability assigns nonnegative, normalised, countably additive weights to events.

DefinitionMathematics & logic · InformationConditional probability and independence

Conditioning restricts attention to an event of positive probability.

DefinitionMathematics & logic · InformationExpected value of a discrete random variable

Expected value weights possible values by their probabilities.

DefinitionMathematics & logic · MeasurementConfidence interval for a population mean

A confidence level describes repeated-sampling coverage of an interval procedure.

MethodInformation · Mathematics & logicBits and unsigned binary encoding

A bit has two possible states; a fixed-width unsigned binary word encodes an integer.

DefinitionInformation · Mathematics & logicShannon entropy of a discrete distribution

Shannon entropy measures uncertainty in a specified probability distribution.

DefinitionInformationAlgorithm and decision procedure

An algorithm specifies effective steps; a decision procedure must terminate on every permitted input.

MethodInformation · Mathematics & logicUndecidability of the halting problem

No total algorithm decides halting for every program and input in a general computation model.

Mathematical statementInformation · Mathematics & logicAsymptotic upper bounds and big O

Big O bounds eventual growth up to a fixed constant factor.

DefinitionMeasurementSI base units

Seven named base units organize the SI quantity system.

DefinitionMeasurementSI decimal prefixes

Prefix symbols specify decimal scale factors.

DefinitionMeasurementMetrological traceability

Connect a result to a reference through documented calibrations.

DefinitionMeasurement · Mathematics & logicFirst-order uncertainty propagation

Combine input uncertainty through a measurement model.

MethodMeasurement · Physics · Mathematics & logicDimensional consistency

Use dimensions to detect impossible quantity equations.

MethodPhysics · Earth & cosmosNewtonian gravitation

An inverse-square force models attraction between point masses.

ModelPhysicsFrequency, wavelength and phase speed

Relate a periodic wave's spatial and temporal repetition.

Definition
DefinitionSource checked

Shannon entropy of a discrete distribution

InformationMathematics & logic

Shannon entropy measures uncertainty in a specified probability distribution.

Key knowledge

For a finite discrete distribution with probabilities pᵢ, Shannon entropy is −Σᵢ pᵢ log₂ pᵢ, measured in bits. A zero-probability term contributes zero by its limiting value.

Values and equations

H(X) = −Σᵢ pᵢ log₂ pᵢ

X
Finite discrete random variable
pᵢ
Probability of outcome i
H(X)
Shannon entropy bits

Conditions and scope

  • pᵢ≥0 and Σᵢpᵢ=1; logarithm base 2 sets the unit to bits.
  • This is distributional uncertainty, not semantic meaning, truth or usefulness.
  • Differential entropy and entropy rates for dependent sequences require separate definitions.
Sources · 7

References

Sources retain their recorded scope. A source link alone does not establish verification of every statement.

  1. A Mathematical Theory of Communication (1948), §6: Choice, uncertainty and entropyClaude E. Shannon, Bell Telephone Laboratories; reprint hosted by Yale UniversityResearch paper · PDFcs.yale.eduRead source
  2. MA261 notes, chapter 27, Definition 145: EntropySiddhartha Gadgil / Indian Institute of ScienceCourse notesmath.iisc.ac.inRead source
  3. John Duchi, Stanford University2 sources
    1. Lecture Notes on Statistics and Information Theory (2023), §2.1.1: EntropyJohn Duchi, Stanford UniversityCourse notes · PDFstanford.eduRead source
    2. Lecture Notes on Statistics and Information Theory (2023), Chapter 11: Zero-term conventionJohn Duchi, Stanford UniversityCourse notes · PDFstanford.eduRead source
  4. A Mathematical Theory of Communication (1948)Claude E. Shannon / Bell System Technical JournalResearch paper · PDFcs.yale.eduRead source
  5. Statistics 311 lecture notes, entropy and relative entropyStanford UniversityCourse notes · PDFweb.stanford.eduRead source
  6. Entropy, Diversity and MagnitudeEmily Roff, University of EdinburghTeaching resource · PDFmaths.ed.ac.ukRead source

Evidence and review

Finite-distribution entropy, base-2 units and the zero-probability convention.

2026-10-10 · Compared Shannon's definition with IISc's displayed formula and Stanford's zero-term convention; retained the leading minus sign missing from extracted original-paper text.

Foundational review

2026-10-10 · Fresh primary-source inspection of the complete compact record, including every statement, equation, quantity and material condition, followed by independent AI editorial review. This check does not certify all linked lessons or every passage of every cited publication.

Sources from other places

Scope or convention differs

Finite-distribution Shannon entropy, with zero terms omitted.

The source leaves its logarithm base unstated and its examples match base 10; the repository explicitly defines base 2 and bits.

Related reference entries

Explore relationships within this area of study and its connections to other areas.

Showing 1–6 of 6 relationships

Scientific relationships · 6 / 6

The relationship type describes the connection. Its linked statements, sources and conditions define the scope.

  1. Uses definition
    Relationship sources · 1

    Linked statements

    • For a finite discrete distribution with probabilities pᵢ, Shannon entropy is −Σᵢ pᵢ log₂ pᵢ, measured in bits. A zero-probability term contributes zero by its limiting value.
  2. Uses definition
    Relationship sources · 1

    Linked statements

    • For finite random variables X,Y, I(X;Y)=Σₓ,ᵧ p(x,y)log₂[p(x,y)/(p_X(x)p_Y(y))]=H(X)−H(X|Y). It is nonnegative and zero exactly when X and Y are independent.
  3. Uses definition
    Relationship sources · 1

    Linked statements

    • For an iid finite-alphabet source, fixed-length block schemes at rates above H(X) bits per symbol can have block error tending to zero. At a fixed rate below H(X), block error tends to one.
  4. Uses definition
    Relationship sources · 1

    Linked statements

    • For finite X,Y, H(X|Y)=Σy p(y)H(X|Y=y) and H(X,Y)=H(Y)+H(X|Y). Conditioning cannot increase this averaged discrete entropy: H(X|Y)≤H(X), with equality exactly for independence.
  5. Uses definition
    Relationship sources · 1

    Linked statements

    • For finite distributions p,q, D(p‖q)=Σx p(x)log₂[p(x)/q(x)]≥0, with equality exactly when p=q. It is asymmetric; positive p at a zero q gives infinite divergence.
  6. Related to
    Relationship sources · 2

    Linked statements

    • Von Neumann entropy is −tr rho log2 rho and equals Shannon entropy of the density operator's eigenvalues. For an ensemble {p-x,rho-x}, accessible classical mutual information from a measurement is bounded by chi=S(sum p-x rho-x)−sum p-x S(rho-x). Nonorthogonal states limit distinguishability. Entanglement and conditional quantum entropy produce behavior absent from ordinary classical conditional entropy; one formula cannot replace every quantum resource capacity.

    The eigenvalue distribution gives a classical entropy expression.

Entry detailsPermanent link to this entry ↗
Record identity
info-shannon-entropy
Revision
core.1
Structured reference record
Essential core · first collection