Attention and the transformer

6 lessons planned · none written yet

1. 1The problem attention solvesNot written yetContext and long-range dependencies.
2. 2Queries, keys and valuesNot written yetAttention as a soft lookup.
3. 3Softmax and weightsNot written yetTurning scores into a mix.
4. 4Multi-head attentionNot written yetMany ways of looking at once.
5. 5The transformer blockNot written yetAttention, MLP, residuals, layer norm.
6. 6Stacking blocksNot written yetWhat layers seem to learn.