Position Matters

Frank and Alex have identical preferences. Without positional encoding, the model can't tell them apart.

1. The problem — attention has no sense of order

Look at how attention computes its output:

Every operation here is row-wise. Q[i] is computed from X[i] alone. The alignment score between person i and person j is Q[i] · K[j] — it only depends on their feature vectors, not where they sit in the sequence.

If you shuffled the rows of X, you'd get a shuffled Q, a shuffled K, a shuffled V — and a shuffled output. The relative scores between any two people would be identical. The model has no way to know that person 1 spoke before person 4.

This is not a bug — it is a direct consequence of how matrix multiplication works. But for language, order is everything.

2. Frank joins the table

We now have four people seated in order:

Frank is new. And he happens to have exactly the same preference profile as Alex — same taste, same priorities, same everything:

Row 1 and row 4 are identical.

Now run them through attention. Since the rows are the same, and W_Q, W_K, W_V are the same matrices applied to both:

Every alignment score involving Frank is identical to the corresponding score involving Alex. Their softmax weights are identical. Their outputs are identical.

The model sees two people at different positions in the sequence — one who spoke first, one who spoke last — and produces the exact same representation for both. Position 1 and position 4 are indistinguishable. Frank might as well not exist.

3. The fix — add a position label to every row

The solution is straightforward: before X enters attention, add a position-specific vector to each row.

P is the positional encoding matrix — same shape as X, one row per position. Each row encodes where in the sequence that person sits. The addition happens element-wise: every feature dimension of every person gets a small positional nudge.

After the addition, Frank and Alex's rows are no longer identical:

Since \(X_{\text{Frank}} = X_{\text{Alex}}\) but \(P_1 \neq P_4\), we get \(X'_{\text{Frank}} \neq X'_{\text{Alex}}\). From this point forward, every downstream computation — Q, K, V, alignment scores, attention weights, output — will differ between Frank and Alex.

Three requirements a good P needs to satisfy:

The original Transformer paper's answer to all three: sine and cosine functions at different frequencies.

4. Designing P — sine and cosine

The positional encoding for position pos and dimension i is:

where d = 6 is the feature dimension, and i = 0, 1, 2 indexes the three sin/cos pairs. Working out each dimension:

Three pairs, three frequencies. Plugging in pos = 1, 2, 3, 4:

5. X' = X + P — Frank and Alex diverge

Adding P to X element-wise:

Frank and Alex started identical. Now compare them directly:

d₀ and d₁ — the high-frequency pair — show the biggest divergence. Frank at pos=1 gets a large positive nudge (+0.841, +0.540). Alex at pos=4 gets a negative nudge (−0.757, −0.654). Their d₀ values go from identical (0.98) to completely different (1.821 vs 0.223).

d₄ and d₅ — barely move. At only four positions apart, the low-frequency pair hasn't had room to diverge yet. Across thousands of tokens, these dimensions would show meaningful separation.

6. What positional encoding changes downstream

X' now flows into attention exactly as X did in Part 1 and Part 2. Using the same W_Q, W_K, W_V, we compute all three projections for all four people.

Q' = X' @ W_Q

Before positional encoding, Frank and Alex would have been identical at [2.885, 0.474]. Now Frank is [5.319, 1.716] and Alex is [2.589, 1.589] — completely different. Daniel's negative d₁ nudge at pos=3 pushes his upscale signal to 3.027 — the strongest upscale query at the table.

K' = X' @ W_K

Frank — massive street-stall broadcast (5.271). His pos=1 nudge amplified everything.
Emma — balanced (2.789 vs 2.538). Her pos=2 nudge is moderate in both directions.
Daniel — strongly upscale (3.908). His pos=3 negative d₁ suppressed street-stall and lifted upscale.
Alex — street-stall dominant (2.565 vs 1.577), same profile as Frank but positionally dampened.

V' = X' @ W_V

Frank — the large positive nudge at pos=1 amplified his info-sichuan signal to 3.963, nearly double the original 2.031 from Part 1.
Daniel — the negative d₁ nudge at pos=3 suppressed info-sichuan (1.626) but strongly amplified info-upscale (3.115).
Alex — the negative nudge at pos=4 pulled info-sichuan down to 2.290 and info-budget down to 0.798.
Frank and Alex now have completely different V' vectors — same knowledge, different positions, different contributions.

Let's look at V specifically, since it carries the actual content that gets blended into the output.

Without positional encoding: Frank and Alex share the same X row, so:

Their content contributions to the output are identical. The model cannot distinguish whose knowledge is whose.

With positional encoding: Frank and Alex now have different X' rows, so:

Since \(P_1 \neq P_4\), we get \(V'_{\text{Frank}} \neq V'_{\text{Alex}}\). The same knowledge, held by two people in different positions, now contributes differently to the final blend.

7. The full picture

Positional encoding is not part of attention itself — it happens before attention, as a one-line preprocessing step. Everything in Part 1 and Part 2 still applies. The only difference is that the input is now X' instead of X.

Two things worth noting before we move on:

Positional encoding is added, not concatenated. It would be natural to think "just append the position as an extra feature." But addition keeps the dimensionality the same — the rest of the architecture doesn't need to change. And because the model learns W_Q, W_K, W_V jointly with the positional signal already mixed in, it learns to disentangle content from position as needed.

The sine/cosine encoding is fixed, not learned. The original Transformer paper used the formula above — no training required. Later models like BERT learned positional embeddings instead, treating each position as a trainable vector. Both approaches work. The sine/cosine version has the advantage of generalizing to sequence lengths longer than anything seen during training.