Prediction, compression and intelligence

Why predicting the next token well requires understanding, and why compression and prediction are the same thing. The core idea behind language models.

6 lessons · 3 written so far

  1. 1The guessing gameStartPlay Shannon’s game and measure how predictable English is.
  2. 2Bits and surpriseStartInformation content, entropy and what “bits per character” means.
  3. 3Compression is predictionStartArithmetic coding: a good predictor is a good compressor.
  4. 4Cross-entropy lossNot written yetThe number every LLM is trained to push down.
  5. 5Why prediction needs understandingNot written yetWhat a model must know to predict the next word of a physics paper.
  6. 6The scaling betNot written yetHutter Prize, Sutskever’s argument and the case against.