Scaled Dot-Product Attention

Every query against every key

Lay the queries out as the rows of a matrix , then do the same with the keys to get . A sentence of five words, each asking its own question, gives five queries and five keys, so twenty-five query-key pairs are waiting to be scored.

One matrix product computes all of them: is a five by five table whose entry in row , column is exactly . Every dot product of the last two chapters, done at once.

Up to now there has been exactly one query, written by hand: what did the cat chase? From here every word carries its own.

So what is a word asking? Nothing that could be written down as a sentence. A query is a vector, not English, and nobody hands the model a list of questions; it learns whatever directions happen to help. What can be described is how a query behaves. Take word (3) chased. It is a verb missing its object, and a query that scores highly against mouse behaves like the question this tutorial has been using all along. The hand-written query was standing in for word 3 the whole time. The difference now is that the other four words get one too.

In the figure above:

1. Query — a row of . Row is the question word is asking.
2. Key — a column of , the same thing as a row of . Column is the label word wears so it can be found.
3. Score — an entry of . Row , column is how well word 's query matches word 's key.

That explains the shading. The single dark cell is one score, : the query of the second word against the key of the fourth, one dot product of the kind the last two chapters were spent on. The shaded row it sits in is all five scores for that one query, its match against every word in the sentence including itself. That row is the list of five numbers the similarity chapter ended with, and every other row is the same list for a different word.

Now apply the scaling. Divide the whole matrix by and run softmax along each row separately, so every row sums to one on its own. Word 2 ends up with its own distribution of attention over the five words, word 3 gets its own, and the matrix as a whole sums to nothing in particular.

The final matrix is a complete set of weights: five rows, each a distribution over the five words. But weights are not an answer. Nothing in that table carries information about the words themselves. It says only how much of each to use, and supplying the content is the job of the values.

The values matrix

Every word hands over a value vector just as it hands over a key, and stacking those as rows gives , one row per word. Multiplying the weights by is the last step, and the one that finally produces something worth reading: row of the result is word 's answer, all five value vectors mixed in the proportions its own row of weights asks for. That is the soft lookup from the dictionary chapter, run five times over, once per word.

Every row of the weights answers one question: when this word builds its new representation, how much of each other word should it pull in?

So the 0.60 where chased meets mouse says that most of chased's answer is made out of mouse's value. Influence has an exact meaning inside this mechanism: the fraction of one word's output vector that came from another word's value. A verb is incomplete without its object, so after this step the representation of chased is no longer chased in the abstract. It is closer to chased a mouse.

The whole mechanism on one line

Everything from the last three chapters fits on one line:

Read it from the inside out and nothing in it is new. scores every query against every key. Dividing by keeps those scores in a range the softmax can work with. The softmax turns each row into weights that sum to one. Multiplying by blends the values in those proportions.

One thing the formula does not say, and it is the question that has been waiting since the dictionary chapter: where do , and come from? They are made from the sentence itself, by three matrices the model learns during training. That is the next chapter.