Information, channels, noise and communication models

Follow the message and the waveform

A source chooses messages from an agreed alphabet. An encoder maps them into transmitted symbols; a physical transmitter produces a waveform. The channel changes that waveform through propagation, filtering, interference or noise. The receiver processes observations and decides which symbols were likely sent. A decoder then reconstructs a message. These stages may be combined in hardware but remain useful conceptual boundaries.

Bandwidth describes a frequency interval, while bit rate counts transmitted bits per second. Neither is the carrier frequency. A carrier can oscillate many times during one information symbol. Distinguish payload, framing and redundancy when quoting a rate: sending more coded bits does not automatically deliver more new information.

Count binary distinctions before using the logarithm

One binary choice has two possible values. Two binary positions can distinguish four combinations, and three can distinguish eight: the count is 2ᵏ for k positions. The base-two logarithm asks the reverse question: which exponent gives this count? Thus log₂2=1, log₂4=2 and log₂8=3. The unit bit connects the information scale to these binary distinctions; it does not mean that every physical signal is a two-level voltage.

For a possible outcome of probability 0<p≤1, the reciprocal 1/p sets the value on that scale. If p=1/8, then 1/p=8 and log₂(1/p)=3 bits. This is the same as −log₂p. The value need not be an integer: p=1/3 gives log₂3≈1.585 bits of self-information, not a fractional binary character in a codeword. For the supplied four-status source below, the individual values are 1, 2, 3 and 3 bits. Multiply each by how often its status occurs to obtain the expected information: 0.5×1+0.25×2+0.125×3+0.125×3=1.75 bits per status. The symbol Σ in the entropy expression means to add these weighted contributions.

Independence matters when joining events. An A followed by a B has probability 0.5×0.25=1/8 under the stated independent source, so it conveys 3 bits: the 1 bit for A plus the 2 bits for B. Multiplication of probabilities becomes addition on the logarithmic scale. For dependent statuses, use their actual joint or conditional probabilities; multiplying the two marginal probabilities can give the wrong result.

Use probabilities without confusing information with importance

For a discrete event of probability p, self-information is−log₂p bits. Its expectation over the source distribution is entropy H=−Σpᵢlog₂pᵢ. This measures uncertainty in that model. A rare trivial status and a rare urgent alarm can have the same mathematical self-information; the number does not measure human importance or truth.

A lossless source code removes predictable redundancy while preserving recoverability for valid messages. A prefix code has no complete codeword that starts another complete codeword, allowing symbol boundaries to be found sequentially. Shorter words can be assigned to more frequent symbols. Expected length and actual record length differ unless the observed proportions match the assumed distribution.

State what the noise model leaves out

An additive model writes the received sample as a noise-free prediction plus a disturbance. A Gaussian model describes a distribution; a white-noise idealization concerns temporal or spectral relationships. Real observations can instead contain bursts, drift, clipping or correlated interference. A model useful for one decision may fail for another.

Signal-to-noise ratio compares signal and noise powers over specified bandwidth and observation conditions. It does not uniquely determine message error without the signalling, timing and detection method. Compression also does not repair channel corruption: source coding and error-control coding solve different problems, even when they operate on the same stream.

A compact four-status source code

Assume independent statuses A, B, C, D with probabilities .5, .25, .125, .125. Use codewords 0, 10, 110, 111. Their weighted mean length is .5×1+.25×2+.125×3+.125×3=1.75 bits/status, equal to this source entropy. A constructed 1000-status record containing 500 A, 250 B, 125 C, 125 D uses 1750 code bits.

A fixed two-bit code uses 2000 bits for that record. The 250 bit difference excludes framing, error protection and metadata needed to agree the code. The prefix tree explains why concatenating words remains recoverable; drawing lengths do not represent channel delay or probability.

Now stipulate equal probabilities for the same four statuses while keeping codewords 0, 10, 110, 111. Each outcome then has 2 bits of self-information and H=2 bits/status. The existing words average (1+2+3+3)/4=2.25 bits/status, while fixed two-bit words average 2. A valid code need not be an efficient code for a changed source. Equality with H in the first example came from that distribution and code, not from a promise that any finite record occupies exactly its length times H.

Try the reasoning

1. Can 1750 bits of compressed status data require more than 1750 channel bits?

   Show the answer and reasoning

   Yes. Framing, error-control redundancy and other protocol overhead can increase transmitted size.

2. Is an event of probability 1 associated with one bit of self-information?

   Show the answer and reasoning

   No.−log₂1=0 bits: that model already predicts the event with certainty.

3. If source probabilities change, is the mean length still 1.75 bits/status?

   Show the answer and reasoning

   Not necessarily. Codewords remain decodable, but their expected lengths must be reweighted by the new probabilities.

4. Keep the same prefix words, but change probabilities of A, B, C, D to 1/8, 1/8, 1/4, 1/2. What are the entropy and expected code length, and does the code stop being recoverable?

   Show the answer and reasoning

   The information values are now 3, 3, 2, 1 bits. Their weighted mean is 3/8+3/8+2/4+1/2=1.75 bits/status. The old word lengths instead average 1/8+2/8+3/4+3/2=2.625 bits/status. The prefix property is unchanged, so valid messages remain recoverable; the short word is now attached to an infrequent status. Mathematical information has not become the length of each actual codeword or a measure of urgency.

What to carry forward

Keep message uncertainty, physical waveform behaviour and delivery performance distinct.