Sampling and inference

What happens when you send a prompt: sampling, temperature, KV caches, context windows and serving costs.

4 lessons planned · none written yet

  1. 1SamplingNot written yetTemperature, top-p and why outputs vary.
  2. 2The KV cacheNot written yetWhy long contexts cost memory.
  3. 3Context windowsNot written yetHow models handle a million tokens.
  4. 4Serving economicsNot written yetTokens per second and dollars per million.