Chapter 02

Attention, per decode step

What a new token needs from the past.

Same five tokens, one forward pass. This time we watch what one attention layer does with them.

01 / 14

The math, once

For one head, with $Q = XW_Q$ , $K = XW_K$ , $V = XW_V$ :

\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}} + M\right)V

$M$ is the causal mask from step 7: $M_{ij} = -\infty$ for $j > i$ and $0$ otherwise, so softmax gives a later key zero weight. Row $i$ of the result only uses row $i$ of $Q$ , but every row $\le i$ of $K$ and $V$ . That asymmetry is the whole next chapter.