Pretraining and scaling laws

Data, compute and parameters: what pretraining is and why bigger kept working.

5 lessons planned · none written yet

  1. 1The pretraining recipeNot written yetTrillions of tokens and one objective.
  2. 2Scaling lawsNot written yetLoss falls as a power law.
  3. 3ChinchillaNot written yetBalancing data and parameters.
  4. 4EmergenceNot written yetAbilities that appear with scale, and the debate.
  5. 5Compute economicsNot written yetWhat a frontier training run costs.