Probability and information

Probability, distributions, Bayes, entropy and cross-entropy: the math under language models.

5 lessons planned · none written yet

  1. 1Probability as countingNot written yetEvents, sample spaces, conditional probability.
  2. 2Bayes’ ruleNot written yetUpdating beliefs with evidence.
  3. 3DistributionsNot written yetDiscrete distributions and expectation.
  4. 4Surprise and entropyNot written yetInformation measured in bits.
  5. 5Cross-entropyNot written yetThe loss function every LLM minimizes.