Masked attention is only the first step. Each layer also thinks alone, stabilizes itself, and passes the result forward.
Everything we have built so far — positional encoding, masked attention — feeds into a single decoder layer. But masked attention is only the first of three steps inside that layer.
More precisely, one decoder layer runs:
Two sub-layers, each wrapped in the same pattern: do something, add the input back, normalize. The input x is X' — the position-aware feature matrix from Part 3. x₂ is the output that flows to the next decoder layer.
GPT-2 (small) stacks 12 of these layers. GPT-3 stacks 96. Each layer sees the output of the previous one — the first layer operates on raw positional embeddings, by layer 96 the representations have been refined 96 times.
We computed this in Part 4. The input is X', the output is:
Each row is a weighted blend of what that person was allowed to hear — only from people who spoke before them. Frank heard only himself. Alex heard mostly Frank.
In a real transformer, W_V is designed so the attention output has the same dimension as the input — d_model. Here our attention output has 3 dimensions while X' has 6. We treat the attention output as already projected back to 6 dimensions before the residual step — exactly as W_O does in Part 2.
Before we add the residual, let's trace every transformation Alex has been through — from raw features all the way through multi-head masked attention to the W_O projection:
Now the residual connection makes clear why it matters: X'_Alex = [0.223, 0.296, 0.305, 1.953, 0.159, 1.080] on its own is small and position-distorted. The attention output [4.814, 7.463, 2.895, 4.667, 4.981, 4.371] carries what Alex learned from the conversation. Adding them together preserves both — Alex's original identity and the new knowledge he gathered.
After masked attention, we add the original input back:
This is the residual connection — also called a skip connection. The input bypasses the sub-layer and gets added directly to its output.
Two reasons this matters:
Gradient flow. In a deep network, gradients can shrink to near-zero as they backpropagate through many layers — the vanishing gradient problem. The residual connection creates a shortcut: gradients can flow directly through the addition, bypassing the sub-layer entirely. This is what makes 96-layer networks trainable.
Preserving what you knew. Attention rewrites each person's representation based on what they heard. But not everything needs to be rewritten. The residual connection says: keep your original features, and add whatever attention taught you on top. You listened to Frank, but you didn't forget who you are.
For Alex:
Alex's vector now contains both his original positional features and what he learned from multi-head masked attention. The addition is literal — the numbers stack.
For all four people:
After the residual addition, we normalize each token's vector independently:
where \(\mu\) is the mean of the vector, \(\sigma\) is the standard deviation, and \(\gamma, \beta\) are learned scale and shift parameters (initialized to 1 and 0). For simplicity we set \(\gamma = 1, \beta = 0\).
Why normalize?
After the residual addition, each token's vector can have values at very different scales. Consider Alex's vector before normalization:
d₁ = 7.790 is the largest, d₂ = 3.186 is the smallest — a difference of nearly 2.4×. Now send this into FFN's first layer x @ W₁. The contribution of each dimension is its value multiplied by its weight. No matter what W₁ learns, d₁ will always contribute ~4× more than d₂ simply because its value is ~4× larger. The weights can't fix a numerical imbalance that large. d₂'s information gets drowned out — not because it's unimportant, but because its value is small.
This is the problem: the magnitude of a value is overriding the significance of a weight. LayerNorm fixes this by pulling all dimensions to the same scale, so W₁'s weights — not the raw values — decide which dimensions matter.
How it works — Alex's full calculation:
Step 1 — Compute mean:
Step 2 — Subtract mean from each dimension:
All values now centered at 0. But 2.233 is still much larger than −0.121.
Step 3 — Square each deviation:
Step 4 — Average the squares (variance):
Step 5 — Take the square root (standard deviation):
The standard deviation 1.413 is the average spread of these six values — a single ruler that describes how much they vary.
Step 6 — Divide each deviation by σ:
Every dimension is now expressed in units of "how many standard deviations from the mean." The relative ordering is unchanged — d₁ is still the highest, d₂ is still the lowest — but the absolute magnitudes are on a common scale. W₁'s weights can finally do their job fairly.
Why per-token? LayerNorm normalizes across the feature dimension — one token's values across all dimensions. BatchNorm normalizes across the batch dimension. For language models, LayerNorm is preferred because each token can be processed independently, regardless of batch size.
For all four people after LayerNorm:
Each row now has mean 0 and standard deviation 1. Ready for FFN.
The second sub-layer is a Feed-Forward Network — two linear transformations with a ReLU activation in between:
Unlike attention, FFN processes each token independently. Every row of x₁ goes through the exact same two weight matrices, but separately — no token looks at any other token here.
What are b₁ and b₂? They are bias terms — one per neuron, added after the linear transformation. W₁ scales and rotates the input, but it always passes through the origin: if the input is a zero vector, the output is also zero. b₁ adds a fixed offset to each neuron, shifting the point at which ReLU activates. Without b₁, a neuron only fires when xW₁ > 0. With b₁, the threshold becomes xW₁ > −b₁ — the model can learn how easily each neuron triggers. b₂ does the same after the second layer. In our example we set b₁ = 0 and b₂ = 0 to keep the arithmetic clean and focus on the core mechanics. In a real trained model both are non-zero learned parameters.
Attention is the conversation — each person gathers information from others. FFN is the thinking that happens after — each person privately processes what they've collected and refines their own representation. No more communication, just computation.
In real transformers, FFN expands the dimension by 4× before compressing back: d_model → 4×d_model → d_model. For our example we use 6 → 3 → 6 to keep numbers manageable.
W₁ and W₂ are FFN-exclusive weight matrices — they appear here for the first time in this series. They have nothing to do with W_Q, W_K, W_V, or W_O from the attention layers. Every sub-layer in a transformer has its own independent set of weights, all learned separately during training.
Computing FFN for Alex step by step:
First layer — x₁ @ W₁:
Apply ReLU — set negative values to 0:
Second layer — ReLU(h) @ W₂:
FFN output for all four people:
This time ReLU kept h₁ and h₂ active for Alex (both positive), zeroing only h₃. Different tokens activate different hidden dimensions — this sparsity is ReLU's contribution to the network's expressiveness.
The same pattern repeats after FFN: add the input back, then normalize.
For Alex:
For all four people:
LayerNorm for Alex:
Final output of one complete decoder layer:
Compare with X' that entered this layer:
Compare with X' that entered this layer — the structure has shifted significantly. d₁ now dominates for all four people, a pattern that emerged from the combination of multi-head attention, residual connections, and FFN processing. The layer has done real work.
x₂ is the output of one decoder layer. In GPT, this becomes the input to the next decoder layer — which runs the exact same sequence of operations.
current input
x⁽⁰⁾ = X′
output
—
what this layer learns
raw features, positional patterns
where \(x^{(0)} = X'\) and L is the total number of layers.
A few things stay constant across all layers:
The weight matrices are different in every layer. W_Q, W_K, W_V, W₁, W₂ are all learned separately per layer. Layer 1's attention weights have nothing to do with layer 12's.
The shape stays the same. Every layer takes d_model dimensions in and produces d_model dimensions out. The architecture is a clean pipeline.
LayerNorm resets the scale after every sub-layer. No matter how large the values get inside a layer, the output is always normalized before passing forward.
Why does stacking help? A single attention layer can capture one level of relationships — who aligns with whom on the surface features. A second layer can capture relationships between those relationships — patterns that only emerge once the first layer has done its work. Deep networks learn hierarchical structure, and language has exactly that: words → phrases → sentences → discourse.
One complete GPT decoder layer, from input to output:
Attention gathers. Residual preserves. LayerNorm stabilizes. FFN refines. Then repeat.
Frank spoke first and anchored the conversation. By the time x₂ emerges from layer 1, his influence has been gathered by attention, stabilized by LayerNorm, and further processed by FFN. In layer 2, the same thing happens again — but now operating on richer representations. By layer 96, the model has refined its understanding of the sequence 96 times. That depth is where the intelligence lives.