Mathematics · Exponents and logarithms · Lesson 1

Powers of two, both ways

A logarithm answers one question: how many times? Rebuild powers of two, run them backwards, and the minus sign in $-\log_2 p$ will stop looking mysterious.

Doubling

$2^n$ means “multiply $n$ twos together.” So $2^3 = 2 \times 2 \times 2 = 8$. Each step up the ladder doubles:

$n$	0	1	2	3	4	5	10
$2^n$	1	2	4	8	16	32	1024

Why is $2^0 = 1$? Walk down the ladder: each step down divides by 2. $8 \to 4 \to 2 \to 1$. Going from $2^1 = 2$ down one more step gives $2 \div 2 = 1$. So $2^0 = 1$: zero doublings leaves you where you started.

What is $2^6$?

$2^5 = 32$, and one more doubling gives $32 \times 2 = 64$.

Below zero

Keep walking down. Every step still divides by 2:

$$2^0 = 1,\quad 2^{-1} = \tfrac12,\quad 2^{-2} = \tfrac14,\quad 2^{-3} = \tfrac18$$

A negative exponent counts halvings. The rule: $2^{-n} = \dfrac{1}{2^n}$. It is never a negative number: $2^{-3}$ is $\tfrac18$, not $-8$.

What is $2^{-4}$?

Four halvings from 1: $1 \to \tfrac12 \to \tfrac14 \to \tfrac18 \to \tfrac1{16}$. Equivalently $2^{-4} = \dfrac{1}{2^4} = \dfrac{1}{16}$. The minus in the exponent means “divide,” not “negative.”

Running it backwards

Exponents answer: start at 1, double $n$ times, where do you land? A logarithm asks the reverse: you landed on $x$. How many doublings did it take?

$$\log_2 x = y \quad\text{means exactly}\quad 2^y = x$$

These are the same fact, read in two directions. $\log_2 8 = 3$ because $2^3 = 8$. Slide through the ladder and watch both sentences change together:

What is $\log_2 32$?

Ask: 2 to what power is 32? $2^5 = 32$, so $\log_2 32 = 5$. Five doublings: $1 \to 2 \to 4 \to 8 \to 16 \to 32$.

What is $\log_2 1$?

2 to what power is 1? $2^0 = 1$, so $\log_2 1 = 0$. You are already at 1, so it takes zero doublings.

Logs of fractions are negative

Now ask the backwards question about a fraction. 2 to what power is $\tfrac18$? From the ladder, $2^{-3} = \tfrac18$. So:

$$\log_2 \tfrac18 = -3$$

Three halvings from 1. The pattern for every positive $x$:

$x > 1$: you double to get there, so $\log_2 x$ is positive.
$x = 1$: no steps, so $\log_2 x = 0$.
$0 < x < 1$: you halve to get there, so $\log_2 x$ is negative.

And $\log_2 0$ has no answer. You can halve forever and get as close to 0 as you like, but you never land on it.

What is $\log_2 0.25$?

$0.25 = \tfrac14 = 2^{-2}$. So $\log_2 0.25 = -2$: two halvings from 1, $1 \to \tfrac12 \to \tfrac14$.

Where the minus sign comes from

A probability $p$ is always between 0 and 1. So by the pattern above, $\log_2 p$ is always zero or negative. That would make a strange measure of surprise, so information theory flips the sign:

$$\text{surprise} = -\log_2 p \text{ bits}$$

Work it through for the question you flagged, $p = 0.25$, one step at a time:

Write $p$ as a power of 2: $0.25 = \tfrac14 = 2^{-2}$.
Read off the log: $\log_2 0.25 = -2$.
Flip the sign: $-\log_2 0.25 = -(-2) = 2$ bits.

$p$	as a power of 2	$\log_2 p$	surprise $-\log_2 p$
1	$2^0$	0	0 bits
$\tfrac12$	$2^{-1}$	$-1$	1 bit
$\tfrac14$	$2^{-2}$	$-2$	2 bits
$\tfrac18$	$2^{-3}$	$-3$	3 bits
$\tfrac1{1024}$	$2^{-10}$	$-10$	10 bits

Read the right-hand column as: the surprise is the number of halvings it takes to get from certainty down to $p$. A one-in-eight event is three halvings away from certain, so it’s worth 3 bits.

Check

How many bits of surprise is an event with probability $\tfrac1{32}$?

$\tfrac1{32} = \dfrac{1}{2^5} = 2^{-5}$.

So $\log_2 \tfrac1{32} = -5$, and $-\log_2 \tfrac1{32} = -(-5) = 5$ bits.

What is $\log_2 1024$?

$2^{10} = 1024$, so $\log_2 1024 = 10$. Worth memorising: 1024 is “about a thousand,” so a thousand-way choice is about 10 bits.

An event’s probability is cut in half. What happens to its surprise?

Halving $p$ is one more step down the ladder: the exponent goes from, say, $-3$ to $-4$. So $-\log_2 p$ goes from 3 to 4. One halving, one extra bit, whatever $p$ was.

What is $\log_2 0$?

$2^y$ is always positive. Halving gets you ever closer to 0 but never there, so $\log_2 x$ heads toward $-\infty$ as $x$ shrinks to 0. In surprise terms: an event you said was impossible ($p = 0$) would be infinitely surprising.

In your own words: why is $-\log_2 \tfrac12 = 1$, and why does the formula need the minus sign?

$\tfrac12 = 2^{-1}$, so $\log_2 \tfrac12 = -1$: one halving from 1. Logs of numbers below 1 are negative, and probabilities are at most 1, so the minus sign turns the answer into a positive count: one halving from certainty is 1 bit of surprise.