Positional Information

The sentence scramble

Feed the model The mouse chased the cat instead of The cat chased the mouse. Because , and act on each row independently, the same words produce the same queries, keys and values. The dot products and the softmax weights move with them, and every word walks away holding exactly the vector it held before, sitting in a different row. Whatever the model understands about this sentence, it has no idea who did the chasing.

The numbers in this tutorial have been quietly breaking too. The embedding chapter said the is fetched twice and both copies come back identical. If that is so, both occurrences hand over the same key, and the query of (3) chased must score them equally — yet every figure gives the first the a weight of 0.04 and the second 0.06. Both cannot be true.

Formally, let be a permutation matrix, so is the sentence with its rows shuffled. The projections are row-wise, so all three shuffle the same way, , , , and the scores come out

The softmax runs along each row and neither permutation changes the set of values inside a row, so it commutes with both. Then cancels against , and the whole mechanism satisfies

Shuffle the input and the output is the old output, shuffled. This is permutation equivariance. The only part of the architecture that treats positions differently at all is the mask, and the mask says which words may be read, never where they sit.

Where order has to come from

Nothing is wrong with the mechanism. The trouble is what contains: row holds the embedding of the word at position and nothing else, so two copies of a word are two identical rows. No arithmetic downstream can separate them, because the dot product and the softmax only ever see the rows they are handed. The information has to arrive with the input.

The most direct arrangement is to write the position into the vector — one extra coordinate holding the index. It costs a single dimension, needs no training, and makes every row unique. It also fails, and so does every obvious repair:

1. The raw index. A score is a sum of products with spread about , so a 64-dimensional model has content scores around . A word at position 500 contributes a term of order 500, and the weights rank by position instead of reading the words.
2. The index divided by the length. Now it lands between 0 and 1, but the third word sits at 0.6 in a five-word sentence and 0.03 in a hundred-word one, and the step between neighbours falls from 0.2 to 0.01. A rule as ordinary as look at the previous word has no fixed numerical form to learn.
3. Multiplying instead of adding. Scaling the embedding by the position makes mouse at position 50 fifty times louder than mouse at position 1, so the score measures volume rather than meaning.
4. One-hot the position. Every position becomes distinct and nothing else: neighbours sit as far apart as strangers, the width of the model depends on the length of the sentence, and a position past has no column at all.

One vector per position

Stop describing a position with a number. Give every position a vector of its own, wide, exactly as wide as the embedding it joins, and add the two. Word (3) chased arrives as its own embedding plus the vector belonging to position 3, and the two copies of the finally differ.

A vector buys what a coordinate could not. Spread over numbers, a position is a pattern rather than a magnitude, so two positions can agree in some coordinates and differ in others, and nothing forces the pattern to grow along the sentence. It can stay small enough to colour a score rather than decide it.

What do those vectors hold? The simplest answer: build a table of shape , fill it with random numbers, and let training sort it out, exactly as for the words themselves. Building becomes two lookups instead of one.

This works, and plenty of large models use it. The catch is the table's size. is fixed before training, and there is no row beyond it — a model built for 1024 tokens does not degrade at token 1025, it has nothing to look up. The last rows also get little practice, since long sequences are rare. And because the rows are independent parameters, the model has to learn that position 3 and position 4 are neighbours from scratch, one pair at a time.

Computing the vectors instead of storing them

Both problems come from storing the vectors. A table has finite capacity, and its rows have no intrinsic relationship. So compute the vector for position from a function instead: no row to exhaust, and the relationship between 3 and 4 is built into the function rather than learned.

One option is binary. Write the position in base 2 and map its bits to coordinates. Each coordinate is 0 or 1, so it never overwhelms the content, every position is unique, and the bits alternate at increasing rates — every step, every two, every four. It is efficient, too: coordinates cover positions where a table of the same width holds .

It also has no gradient. Jumping between 0 and 1 leaves no slope, and the distances are erratic: 7 () and 8 () are neighbours yet differ in every coordinate, while 0 () and 8 differ in one bit despite being eight steps apart. Nearness in the sentence and nearness in the encoding come apart.

Keep what worked — coordinates changing at different rates — and drop the jumping. A flipping bit is a square wave, a continuous wave forced into corners. Replace each bit with a smooth wave of the same wavelength and the coordinate glides between and instead of snapping.

Sinusoidal positional encoding

Group the coordinates into pairs. Each pair holds a sine and a cosine turning at the same frequency:

Here is the absolute position and runs to . There are no trainable parameters: the vector for position 5, or 5,000,000, is computed on demand.

The real test is not range but relative distance. Language runs on relations like the adjective just before this noun, and a query at position 10 looking back at position 7 should be doing the same thing as a query at 40 looking back at 37. The model should learn the relation once, not once per position.

That operation exists, and it is why the formula pairs sines with cosines. Shift a pair from to :

Every entry of that matrix is built from the offset alone. The starting point does not appear. Moving three steps is the same small rotation whether it starts at word 7, 37 or 3997.

Rotary position embedding

Notice what has and has not been shown. A shift is a fixed rotation, so a relative offset is something the model can express. Nothing makes it do so. The positional vector is added to , and only then does meet and , so the rotation is something those two learned matrices receive and are under no obligation to pass on. The structure is a property of the input, not a promise about the score.

The scheme that took over keeps the frequencies and changes where they act. Let and work first, then rotate the query and the key by an angle set by their positions. A query at position becomes , where turns each pair by its own angle . A rotation leaves a dot product alone, and turning by then is turning by :

The starting positions cancel. What survives is the gap. Relative position is no longer something the model is merely able to represent; it is the only thing the score can depend on.

Two things come free. The rotation happens after the projections, so is never touched and a value goes back to being purely what the word means. And a rotation preserves length, so the positional signal can never grow loud enough to drown the content — which is exactly what went wrong when the index was stored in a coordinate.

What this finally explains

This chapter opened with two claims that could not both hold. The two copies of the come back from the embedding table identical, and every figure gives them weights of 0.04 and 0.06. Both are true now: the rows leave the table identical and do not stay that way, because something that depends on where they sit is folded in before attention sees them.

The scramble is settled too. The cat chased the mouse and The mouse chased the cat no longer produce the same vectors in a different order, because cat at position 2 and cat at position 5 are no longer the same row going in. The mechanism never changed. It simply stopped being handed a bag of words.

What sits around attention — the residual stream, the normalisation, the feed-forward layer — is the next chapter.