The Shape of the Space

Three ways to say two vectors are close

Every chapter so far has ended with a table of vectors, one row per word, and a claim that words which behave alike end up close. "Close" has been doing a lot of work without being pinned down. There are three common ways to turn two rows into one number, and they disagree more often than you would expect.

The dot product multiplies the rows column by column and adds the results. The Euclidean distance subtracts one row from the other and measures the length of what is left. Cosine similarity, which the distributional chapter used to get 0.926 for king and queen, is the dot product divided by both lengths. Run all three on the three count rows from that chapter:

On these three pairs the measures happen to agree about the order: king and queen are the closest pair, queen and apple the furthest. Two of them are similarities, where bigger means closer, and one is a distance, where smaller means closer, but the ranking is the same. That agreement is luck: the three measures react differently to how long a row is, and the lengths of these rows are not an accident.

Each measure answers a different question. Distance asks how far apart two points are. Cosine asks whether two arrows point the same way, ignoring how long they are. The dot product asks both at once: it is the cosine multiplied by the two lengths, so it rewards pointing the same way and it rewards being long.

Length is mostly frequency

The length of a row, its norm, is the square root of the dot product of the row with itself. For the three count rows it is for king, for queen and for apple. Nothing about royalty or fruit decides those numbers. They count how many times each word turned up next to the eight context words, which is to say how often the word was used.

That is easy to see by changing only the frequency. Write every sentence about the king out twice. Nothing about how king is used has changed; the corpus just mentions him more. The row doubles, every count going from 1 to 2 and from 0 to 0:

Cosine does not move. The dot product doubles and the distance more than triples, although the word means exactly what it meant before.

The dot product has a stranger failure hiding in the original table. queen scores 6 against itself and 6 against king: by dot product, queen is exactly as similar to king as it is to queen. With a longer row for king it would be more similar to king than to itself. A measure where a word is not its own nearest neighbour is measuring something other than resemblance.

Trained embeddings keep the same habit. Schakel and Wilson found that the length of a word2vec vector is tied to how often the word occurs and how consistently it is used: words seen once or twice keep short vectors, and so do the very commonest words, whose contexts are so varied that their updates pull in every direction. So in a raw embedding table, direction carries the meaning and length carries mostly how often the word was seen. The standard move is to divide every row by its norm before comparing anything. Once every row has length 1, the three measures stop disagreeing, because the squared distance between two unit vectors and depends only on their cosine:

The distance between unit rows is a decreasing function of the cosine, and the dot product of unit rows is the cosine itself. On the normalised count rows the distances come out 0.385, 1.171 and 1.414 for the three pairs, which is exactly , and . After normalisation there is one ranking, not three.

Directions that mean something

Closeness is one number per pair. The claim that made word2vec famous is about something more: that the difference between two vectors can mean something, and mean the same thing in different places. The offset from man to woman is supposed to be roughly the offset from king to queen, so that the four words form a rough parallelogram.

The ten sentences cannot show this: they never mention a man or a woman. So the rest of this chapter uses a small table for the five-word vocabulary of the one-hot chapter, five columns wide, written by hand to behave the way trained vectors do. No column is a clean feature; each word is a mixture.

The two offsets are and . They share the step down in column 3, but queen also has something of its own in column 5 that woman does not. The cosine between the two offsets is 0.743: the same direction, roughly, and not exactly. That is what real embeddings look like. The parallelogram is never a parallelogram, only something leaning towards one.

What king − man + woman ≈ queen really measures

The analogy test turns the parallelogram into a question. Start at king, subtract man, add woman, and look for the row nearest to where you land. Written with = man, = king and = woman, the answer is the word whose unit row has the highest cosine with the target. This method is called 3CosAdd:

Two things in that line matter more than the arithmetic. The first is the condition under the : the three input words are not allowed to be the answer. Run the search on the toy table without that condition and the ranking comes out like this:

The target is closest to king itself. Subtracting man and adding woman moves the point, but not far enough to leave king behind, and this is the normal situation with real vectors too. Linzen showed that on standard analogy sets, baselines that ignore the offset altogether, such as returning the nearest neighbour of with the inputs excluded, already get a substantial share of some categories right, and Nissim, van Noord and van der Goot showed how often the excluded input would otherwise have won. The exclusion rule is not a detail. It does a good part of the work.

The second thing is what the cosine with the target adds up to. Because every row has been normalised, the target's length is the same for every candidate, and ranking by the cosine is the same as ranking by a sum of three cosines:

So an analogy is not a search along a meaning direction; it is a vote between three similarities: be like king, be unlike man, be like woman. For the toy table the three terms are:

king wins on the first term by a margin, 1.000 against 0.850, that the other two terms cannot make up. queen wins the analogy only because it is very like king to begin with, and slightly more like woman and less like man than king is. The gender offset is a small correction on top of a large similarity.

That also explains why the result is fragile. A different scoring rule, 3CosMul from Levy and Goldberg, multiplies the three similarities instead of adding them, so one large term cannot drown the others; on this table it puts queen first even without the exclusion, 0.955 against 0.951 for king. And small changes to one row flip the answer. Lower queen's private fifth column from 1.0 to 0.6 and queen scores 0.977 and beats king outright, exclusion or not; raise it and queen slides further behind. Nothing about gender changed between those tables. What changed is how much queen has that king does not.

None of this means the offsets are empty. The two differences in the table do point roughly the same way, and in large trained models many such offsets are consistent enough to be useful. It means a single analogy is a weak instrument: it measures a sum of similarities, under a convention that hides the most common answer, and a reported success can come from the geometry you hoped for or from the words around it.

Nearest neighbours are one matrix product

Almost everything done with an embedding table, from the analogy test to finding related words, reduces to one operation: given a vector, find the rows closest to it. With every row normalised, that is a single matrix-vector product followed by a sort. The table times the query gives one cosine per word; the largest are the neighbours.

import numpy as np

def neighbours(table, words, query, k=5):
    """Rows of `table` with the highest cosine to `query`."""
    unit = table / np.linalg.norm(table, axis=1, keepdims=True)
    scores = unit @ (query / np.linalg.norm(query))
    best = np.argsort(-scores)[:k]
    return [(words[i], round(float(scores[i]), 3)) for i in best]

On a vocabulary of 50,000 words with 300 columns, one query costs 15 million multiply-adds, which a laptop does in milliseconds. For millions of rows, as in search over documents, exact search gets expensive and systems switch to approximate indexes that check only a promising fraction of the rows and occasionally miss a true neighbour. The quantity they approximate is still the cosine from this chapter.

All of it rests on an assumption that has gone unexamined since the first chapter: that the word you are looking up has a row. Ask for the neighbours of kingdoms, or of qeen typed in a hurry, and the table has nothing to multiply. The next chapter is about that gap, and about why modern tables are built out of pieces of words rather than whole ones.