Attention, But Make It Multi-Head

One question isn't enough. Alex, Emma, and Daniel ask two at once.

1. Why one head isn't enough

In Part 1, Alex asked one question: "Where can I get something spicy, fast, and cheap?"

The attention mechanism answered it well. It computed Q, matched it against K, and blended V accordingly. Alex ended up drawing mostly from his own knowledge, Daniel barely registered.

But consider what Alex actually needs to decide where to eat.

He has two concerns, not one. He cares about taste — spicy, fast, bold flavors. And he cares about budget — he's not going to a place Daniel would recommend, because Daniel's idea of dinner costs three times what Alex wants to spend.

A single attention head can only learn one projection. One W_Q, one W_K, one set of alignment scores. If the head learns to align on taste, it ignores budget. If it aligns on budget, it loses taste.

This is the problem multi-head attention solves.

Instead of one head asking one question, multiple heads ask different questions simultaneously — each with its own W_Q, W_K, W_V — and their answers get combined at the end.

In our case: two heads, two questions.

2. Two heads, two questions

Each head is a complete, independent attention computation. The input X is the same — Alex, Emma, and Daniel's preference vectors — but each head projects it differently.

Head 1 uses \(W_Q^1, W_K^1, W_V^1\) — learned weights that project X into taste and vibe space. Head 2 uses \(W_Q^2, W_K^2, W_V^2\) — a completely separate set of weights that project X into budget space.

The two heads never share weights. They train independently, learn independently, and attend independently. The only thing they share is the input.

3. Head 1 — taste and vibe

Head 1 is identical to what we computed in Part 1. The weight matrices project X into intent/knows/info space:

The full derivation is in Part 1. The results:

Head 1 tells us: Alex's street-stall craving matched his own Sichuan knowledge — 92.9% self-attention. Daniel's upscale craving matched his own upscale knowledge — 74.4% self-attention. Emma sits between both, drawing from each.

But Head 1 knows nothing about what things cost.

4. Head 2 — budget

Head 2 asks a different question entirely. The weight matrices project X into budget space — who's asking about affordable places, and who's asking about high-end experiences.

W_Q² — budget₁ captures low-budget intent (spicy, cheap, fast), budget₂ captures high-budget intent (cozy, flavor, vibe)
W_K² — knows_low signals knowledge of affordable places, knows_high signals knowledge of premium places
W_V² — price_info carries concrete budget knowledge, experience_info carries high-end experience knowledge

Q² = X @ W_Q²

K² = X @ W_K²

V² = X @ W_V²

Alex Q²=[2.777, 0.577] — asking firmly in the low-budget direction. V²=[2.797, 0.607] — his knowledge is mainly about affordable places.
Emma Q²=[1.239, 1.261] — balanced between both budget ranges.
Daniel Q²=[0.646, 2.742] — asking firmly in the high-budget direction. V²=[0.655, 2.757] — his knowledge is mainly about premium experiences.

S² = Q²Kᵀ² / √2

Softmax → A²

Compare A² with A¹:

Daniel's self-attention jumps from 0.744 to 0.897 in Head 2 — on budget, he is even more isolated. Nobody else is in his league. Alex stays similarly self-referential. Emma remains the most balanced in both heads.

Output² = A² @ V²

Head 2 captures something Head 1 completely missed: Alex and Daniel are on opposite ends of the budget spectrum, and Emma is exactly in the middle.

5. Concatenate and project

Each head has produced an output matrix. Now we combine them.

The combination is not averaging, and not adding — it is concatenation. Each person's two output vectors are placed end to end, forming a single longer vector:

This 5-dimensional vector now contains everything — taste, vibe, and budget — all in one place. But it's in the wrong shape to feed into the next layer, which expects the original 6-dimensional space. So we project it back down with W_O ∈ ℝ⁵ˣ⁶, a learned weight matrix that decides how to blend the two heads' contributions:

Each row maps one concatenated dimension back into the original feature space. info-sichuan flows strongly into spicy and fast. info-upscale flows into cozy and vibe. price_info flows into cheap. experience_info flows into flavor and vibe.

Final Output = Concat @ W_O

Let's expand Alex's row explicitly to see how W_O combines both heads:

Notice why cheap = 4.001 is so dominant. Two terms drive it:

Head 1's budget signal and Head 2's price signal both flow strongly into cheap via W_O — two heads, two angles, both pointing at the same dimension. W_O is where those contributions stack. This is the payoff of multi-head attention: independent views of the same input, combined into a richer final representation than any single head could produce alone.

Alex [2.357, 4.001, 1.122, 2.285, 2.480, 1.667] — cheap dominates. Head 1 said he craves street-stall food; Head 2 said he's low-budget. W_O amplified both signals into a strong cheap output.
Emma [1.634, 3.130, 1.589, 1.568, 3.215, 2.916] — flavor and cheap lead. Both heads found her balanced between the two extremes, and the final output reflects that.
Daniel [1.249, 2.459, 2.161, 1.180, 4.047, 4.301] — flavor and vibe dominate. Head 1 said he craves atmosphere; Head 2 said he's high-budget. W_O wove both signals into a strong flavor+vibe output.

6. The full picture

Multi-head attention is not a fundamentally different operation from single-head attention. It is the same operation, run in parallel, multiple times, each time asking a different question — then combining the answers.

For each head \(h = 1, 2, \ldots, H\):

Then concatenate and project:

In our example, H = 2. In GPT-3, H = 96.

What each piece does:

Multiple W_Q, W_K, W_V — each head learns to ask a different question. No coordination between heads during training; they find their own niches.

Independent softmax per head — each head produces its own attention distribution. Head 1 might attend broadly; Head 2 might attend sharply. They don't interfere.

Concatenation — preserves everything. No information from either head is discarded before W_O sees it.

W_O — the mixer. It learns how to combine the heads' contributions into a coherent final representation. This is where "taste matters more than budget in this context" gets decided.