Three people are trying to pick a restaurant tonight. Each has their own preferences, and each knows something about the options. The question is: how do you combine everyone's knowledge so that each person ends up with the recommendation that fits them best? That's exactly what attention does.
Alex, Emma, and Daniel are going out for dinner tonight. Each person carries a feature profile — a set of traits that characterize who they are and what they care about. These features aren't just preferences. As we'll see, the same profile will be decoded in three different ways: what they're craving, what restaurants they know about, and what specific information they're holding.
We encode each person as a vector across six dimensions — [spicy, cheap, cozy, fast, flavor, vibe]. This is embedding — compressing who someone is into numbers. The same vector X will later be projected three ways to extract three different kinds of meaning:
Each row is one person. Each column is one feature. The same X is the starting point for Q, K, and V — three decoders, three different readings of the same profile.
The same feature profile X gets run through three different decoders — Q, K, and V. Each decoder is specialized for a different kind of reading. They don't see different people; they see the same person differently:
Three decoders, three specialized readings of the same profile. Each one extracts something different from the same X.
Q vs K vs V in plain terms: Q asks "what am I craving?" — it drives the search. K says "I know this type of place" — it's the label that gets matched against Q. V says "here's the actual info" — the restaurant name, the dish, the wait time. Q and K play the matching game; V delivers the goods.
WQ ∈ ℝ6×2 — a demand decoder: given your feature profile, it estimates how strongly you're craving each type of experience
Each row is an input feature. Each column is a query direction. spicy contributes to street-stall ("I want it spicy, fast, and cheap"), cozy contributes to upscale ("I want good atmosphere and comfort").
street-stall — cares about spice, price, speed — nothing else matters
upscale — cares about vibe and coziness — price and heat are irrelevant
WK ∈ ℝ6×2 — a knowledge decoder: given your feature profile, it decodes what kinds of places you're likely to know about
WK and WQ are deliberately different. "Craving bold food" and "knowing where bold food is" are two different things — the model needs separate weights to learn them independently.
knows-street-stall — knows where to find spicy, cheap, fast places
knows-upscale-flavor — knows upscale spots with great atmosphere and genuine flavor
WV ∈ ℝ6×3 — a content extractor: given your feature profile, it pulls out the actual knowledge you're holding — no fluff, just the substance
V has three output dimensions — it carries content, not a matching signal.
info-sichuan — knows Sichuan-style spots: spicy, fast — which restaurants, how long they've been open, what to order
info-budget — knows budget-friendly spots with real flavor: cheap eats that actually taste good
info-upscale — knows the upscale scene: cozy, great vibe, places worth the price
Now we run X through each decoder to extract what each person is craving, what they know, and what content they're holding.
Q = X WQ — "you have these features, let me decode what you're craving and how strongly"
Alex [2.885, 0.474] — W_Q decoded his profile (spicy↑ cheap↑ fast↑) into a strong street-stall craving. He didn't say it out loud — his features gave it away.
Emma [1.293, 1.210] — balanced craving in both directions. Her mixed profile (cheap↑ cozy↑) doesn't pull strongly either way.
Daniel [0.742, 1.921] — W_Q decoded his profile (cozy↑ flavor↑ vibe↑) into a strong upscale craving. He wants atmosphere and quality, not speed or heat.
K = X WK — "you have these features, let me infer what knowledge you're carrying"
Alex [2.850, 0.610] — W_K decoded his profile (spicy↑ cheap↑ fast↑) into strong street-stall knowledge. People like him tend to know exactly which stalls are worth it.
Emma [1.260, 1.338] — balanced knowledge signal. Her mixed profile suggests she has something useful in both directions.
Daniel [0.701, 2.796] — W_K decoded his profile (cozy↑ flavor↑ vibe↑) into deep upscale-flavor knowledge. He's the one to ask about atmosphere and quality.
Note: Alex's Q and K are both strong in the first dimension — a near-coincidence of these numbers. In a real model, WQ and WK are trained separately and almost never produce the same result. That is precisely why two matrices are needed.
V = X WV — "you have these features, let me extract the actual content you're carrying"
Alex [2.031, 1.303, 0.479] — W_V extracted mostly Sichuan content from his profile: which stalls, how long open, what to order. His features don't carry much upscale substance.
Emma [0.404, 1.217, 1.189] — W_V extracted a balanced content mix: some budget-flavor knowledge, some upscale. She's a useful source in both directions.
Daniel [0.526, 1.320, 1.949] — W_V extracted mostly upscale content: the cozy spots, the vibe, which places are worth the price. Little Sichuan substance in his profile.
For each person, we take their decoded craving (Q) and score it against every person's decoded knowledge (K) using the dot product. The higher the overlap between what one person is craving and what another person knows, the higher the score.
S[i][j] = Q[i] · K[j] answers: how well does person i's craving align with what person j knows?
Expanding each pair explicitly:
These 9 dot products can be computed in one step using matrix multiplication. Q is 3×2 and K is also 3×2 — they can't be multiplied directly. Transposing K to 2×3 makes the shapes compatible, and Q @ Kᵀ batches all 9 dot products at once:
Before softmax, we divide by \(\sqrt{d_k}\) where \(d_k = 2\) is the dimension of Q and K. This prevents the dot products from growing too large and making softmax overly sharp:
Alex → Alex = 8.511 → 6.018 — his intent (street-stall=2.885) strongly matches his own knowledge (knows-street-stall=2.850).
Alex → Daniel = 3.347 → 2.367 — some overlap, but Alex's street-stall doesn't align well with Daniel's knows-upscale-flavor-dominant profile.
Daniel → Daniel = 5.891 → 4.165 — Daniel's intent (upscale=1.921) strongly aligns with his own knowledge (knows-upscale-flavor=2.796).
We now apply softmax to each row of S. Each row represents one person's scores across all others — softmax turns these into attention weights: how much should I draw from each person's knowledge?
Alex's row: [6.018, 3.018, 2.367]
Emma's row: [3.127, 2.297, 3.032]
Daniel's row: [2.325, 2.479, 4.165]
Collecting all three rows into a matrix:
Alex — scores [6.018, 3.018, 2.367] have a large gap, so he draws mostly from himself (0.929).
Emma — scores [3.127, 2.297, 3.032] are close, so she draws roughly equally from Alex (0.426) and Daniel (0.388).
Daniel — scores [2.325, 2.479, 4.165] heavily favor himself (0.744).
The softmax weights tell us how much to trust each person. We apply those weights to each person's V — their actual content — and sum the result:
Expanding each row explicitly:
Alex [1.909, 1.270, 0.533] — output dominated by info-sichuan (spicy/fast). He paid 92.9% attention to himself, and his own V is strongest in info-sichuan.
Emma [1.142, 1.292, 1.068] — output spread across all three dimensions. She split attention between Alex and Daniel, whose V vectors complement each other.
Daniel [0.697, 1.310, 1.666] — output dominated by info-upscale (cozy/vibe). He paid 74.4% attention to himself, and his own V is strongest in info-upscale.
The output is a weighted blend of everyone's real information. Nobody is ignored entirely — even the lowest-weight person still contributes a little. This is why it's called soft attention, not hard selection.
V is the first decoding — each person's raw content extracted from their profile. Output is the second — the same content, redistributed by attention weights.
V — raw content, before blending:
Output — content after attention reweighting:
Alex — output stays close to his own V (92.9% self-attention). His craving aligned so strongly with his own knowledge that he barely needed anyone else.
Emma — output shifts noticeably. She drew from both Alex and Daniel, so her final content is more balanced than her raw V.
Daniel — output stays close to his own V (74.4% self-attention). His upscale craving matched his own upscale content.
Attention doesn't invent anything. It just decides how to mix what's already there.
Alex ends up with a recommendation dominated by Sichuan info — his craving aligned so strongly with his own knowledge that attention let him mostly listen to himself. Daniel gets the upscale recommendation for the same reason. Emma, with a balanced profile, draws from both sides and gets a more even blend.
Nobody was ignored. Nobody dominated unfairly. Attention allocated influence in proportion to alignment — three people, one shared pool of knowledge, each walking away with the recommendation that fit them best.