The log rules

Counting twos

Multiply two powers of 2 and count the twos:

$$2^3 \times 2^2 = (2 \times 2 \times 2) \times (2 \times 2) = 2^5$$

Three twos and two more make five twos. In general, multiplying powers adds the exponents: $2^a \times 2^b = 2^{a+b}$. It works for negative exponents too: $2^3 \times 2^{-1} = 8 \times \tfrac12 = 4 = 2^2$, and $3 + (-1) = 2$.

The product rule

Now read the same fact as logs. Say $x = 2^a$ and $y = 2^b$, so $\log_2 x = a$ and $\log_2 y = b$. Then:

1. $xy = 2^a \times 2^b = 2^{a+b}$
2. So $\log_2(xy) = a + b$
3. Substitute back: $\log_2(xy) = \log_2 x + \log_2 y$

Example: $\log_2(8 \times 4) = \log_2 32 = 5$, and $\log_2 8 + \log_2 4 = 3 + 2 = 5$.

On a log scale, multiplying is laying rods end to end. Each rod’s length is a log. Pick two numbers (they needn’t be powers of 2) and see where the second rod ends:

That is how a slide rule multiplies: two rulers marked on a log scale, slid along each other so their lengths add.

Why surprises add

Two independent events: a coin lands heads ($p = \tfrac12$) and a four-sided die shows 1 ($p = \tfrac14$). The chance of both is the product, $\tfrac12 \times \tfrac14 = \tfrac18$. Now the surprise, step by step:

1. Both together: $-\log_2 \tfrac18 = 3$ bits.
2. Separately: $-\log_2 \tfrac12 = 1$ bit and $-\log_2 \tfrac14 = 2$ bits.
3. $1 + 2 = 3$. The product rule guarantees this always works.

That’s the reason information is measured with a log. Probabilities of independent events multiply, but we want amounts of information to add, the way lengths and weights do. The log converts one into the other. It’s also why a language model’s loss for a whole sentence is just the sum of the per-token surprises.

The quotient rule

Dividing powers subtracts exponents: $2^a \div 2^b = 2^{a-b}$ (cancel $b$ twos from top and bottom). So, by the same three steps as before:

$$\log_2 \frac{x}{y} = \log_2 x - \log_2 y$$

Example: $\log_2 \tfrac{32}{4} = 5 - 2 = 3$, and indeed $\tfrac{32}{4} = 8$.

Set $x = 1$ and you get a useful special case. Since $\log_2 1 = 0$:

$$\log_2 \frac{1}{y} = 0 - \log_2 y = -\log_2 y$$

So surprise has a second, friendlier form: $-\log_2 p = \log_2 \dfrac1p$. A “1 in 8” event is $\log_2 8 = 3$ bits. No minus sign needed: just ask how many doublings it takes to reach the “1 in …” number.

The power rule

Raising a power to a power multiplies exponents: $(2^a)^k$ is $k$ copies of $2^a$ multiplied together, which is $2^{a + a + \dots + a} = 2^{ak}$. So:

$$\log_2(x^k) = k \log_2 x$$

Example: $\log_2(8^2) = 2 \times \log_2 8 = 2 \times 3 = 6$, and $8^2 = 64 = 2^6$.

This is how you count bits in a string. A random 10-character string over 27 symbols has $27^{10}$ possibilities. Its information is $\log_2(27^{10}) = 10 \log_2 27 \approx 10 \times 4.75 = 47.5$ bits: 4.75 bits per character, times 10 characters.

Check