How LLMs work · Prediction, compression and intelligence · Lesson 2

Bits and surprise

Last time you measured surprise one guess at a time. Now the averaged version, entropy: the single number that says how unpredictable a source is, and the floor no compressor can beat.

Surprise, recapped

An outcome you gave probability $p$ carries $-\log_2 p$ bits of surprise. Probability $\tfrac12$ is 1 bit, $\tfrac14$ is 2 bits, $\tfrac18$ is 3 bits: every halving adds one bit. Something certain ($p = 1$) carries $-\log_2 1 = 0$ bits. You learn nothing from it.

How many bits of surprise is an outcome with probability $\tfrac18$?

$\tfrac18 = 2^{-3}$, so $-\log_2 \tfrac18 = -(-3) = 3$ bits.

Entropy: surprise on average

Before you see the outcome, you can ask: how surprised should I expect to be? Weight each outcome’s surprise by how often it happens and add up:

$$H = \sum_{\text{outcomes}} p \cdot (-\log_2 p)$$

That is the entropy, in bits. Try it on a coin. Slide the chance of heads:

A fair coin has the most entropy: exactly 1 bit. A coin that lands heads 99% of the time is mostly boring. Its rare tails is very surprising (6.6 bits), but it’s so rare that the average is only about 0.08 bits.

A coin lands heads 90% of the time. Its entropy is…

Tails costs $-\log_2 0.1 \approx 3.32$ bits but happens only 10% of the time. Heads costs $-\log_2 0.9 \approx 0.15$ bits. Average: $0.1(3.32) + 0.9(0.15) \approx 0.47$ bits. Lopsided means predictable means less entropy.

A worked example

Tomorrow’s weather, as your forecast sees it: sun $\tfrac12$, cloud $\tfrac14$, rain $\tfrac14$.

Sun: $\tfrac12 \times (-\log_2\tfrac12) = \tfrac12 \times 1 = 0.5$
Cloud: $\tfrac14 \times (-\log_2\tfrac14) = \tfrac14 \times 2 = 0.5$
Rain: $\tfrac14 \times 2 = 0.5$

Add them: $H = 0.5 + 0.5 + 0.5 = 1.5$ bits. And here’s what that means concretely: give sun the code 0, cloud 10, rain 11. On average a day costs $\tfrac12(1) + \tfrac14(2) + \tfrac14(2) = 1.5$ bits. The entropy is the length of the best possible code.

Four outcomes with probabilities $\tfrac12, \tfrac14, \tfrac18, \tfrac18$. What is the entropy in bits?

Surprises: 1, 2, 3, 3 bits.

Weighted: $\tfrac12(1) + \tfrac14(2) + \tfrac18(3) + \tfrac18(3) = 0.5 + 0.5 + 0.375 + 0.375 = 1.75$ bits.

Shape a distribution

Four kinds of weather. Drag the weights (they’re rescaled to add to 1) and watch the entropy. Can you get it to 2 bits? To 0?

Entropy is highest when every outcome is equally likely ($\log_2 4 = 2$ bits for four), and zero when one outcome is certain. Everything in between measures how much you don’t know.

Which distribution over 8 outcomes has the highest entropy, and how much?

Uniform is maximal, and for $n$ equal outcomes $H = \log_2 n$. $\log_2 8 = 3$ because $2^3 = 8$.

Bits per character, for real text

Treat text as a source that emits one character at a time. If you ignore context and only know how common each letter is, the entropy is the bits per character you’d need. Type or paste anything:

Letter counts alone get English to about 4.1 bits per character, since e and space are common and q and z rare. Shannon’s guessers reached about 1 bit. The gap, roughly 3 bits per character, is what context buys: knowing the previous letters, words and meaning. Capturing that context is the entire job of a language model.

Why do human guessers beat the letter-frequency estimate by so much?

After “th” the next letter is very likely e; after “the capital of France is ” it’s almost surely P. The probabilities that matter are conditional on what came before, and those are far more lopsided, so the average surprise is far lower.

Bits or nats

Machine-learning code uses the natural log, $\ln$, instead of $\log_2$. The unit is then the nat. It’s the same quantity in a different unit, like miles and kilometres: 1 nat $= \frac{1}{\ln 2} \approx 1.44$ bits. When a training run reports a loss of 2.0, that’s 2.0 nats of average surprise per token, about 2.9 bits.

Check

A 1,000-character text is compressed to 1.2 bits per character. How many bytes is that? (8 bits = 1 byte.)

$1000 \times 1.2 = 1200$ bits. $1200 \div 8 = 150$ bytes. Stored plainly, at one byte per character, it would be 1,000 bytes.

A source always emits the letter a. What is its entropy in bits?

One outcome with $p = 1$: $1 \times (-\log_2 1) = 1 \times 0 = 0$. A perfectly predictable source carries no information, and you could “send” it in zero bits.

A model’s training loss falls from 3.0 to 2.5 nats per token. What happened?

The loss is the average of $-\ln p$ for the token that actually came next. Lower loss means higher probability on the right answers. In bits it went from about 4.3 to 3.6 per token.

In your own words: what does entropy measure, and when is it largest?

It is the average surprise, $\sum p(-\log_2 p)$: how many bits per symbol you need on average, or equivalently how unpredictable the source is. It is largest when all outcomes are equally likely ($\log_2 n$ bits) and zero when one outcome is certain.