Number Class
← Back to Explore
Computer Science Artificial Intelligence & Data Natural Language Processing

Meaning Isn't a Place in the Model — It's a Ruler You Chose

A 2023 paper shows cosine similarity needs a special 'causal inner product' before distance means anything at all; without it, tulips and recessions score 0.68 similar

S.J. Nam 8 min read
Meaning Isn't a Place in the Model — It's a Ruler You Chose

The Tulip Problem

Ask an OpenAI embedding model to compare "Recession is happening in the economy" with a sentence about tulip biology, and you'd expect two unrelated ideas to sit far apart in the model's internal space. They don't. A developer working with the text-embedding-ada-002 model found the cosine similarity between the two floating persistently above 0.68 — high enough that, on paper, tulips and recessions look like close cousins.
"I'm encountering an issue where the cosine similarity between embedding vectors extracted using the text-embedding-ada-002 model consistently yields values above 0.68, even for dissimilar or unrelated texts."
That number is the whole essay in miniature. We've been sold a tidy story about ChatGPT's insides: words become points, meaning becomes distance, and famous arithmetic like "king minus man plus woman equals queen" proves the machine has found some Platonic geometry of language. It hasn't. The geometry inside these models is not a landmark we've discovered so much as a ruler we've chosen — and the choice of ruler quietly decides what we're allowed to call "meaning" at all.

The Trick Behind the Trick

Start with the demo everyone reaches for. Tomás Mikolov and colleagues showed in 2013 that word2vec embeddings could complete analogies through simple vector subtraction and addition, and the result — king minus man plus woman lands near queen — became the founding parable of the whole field. But the parable has a footnote most retellings drop. As data scientist Florian Huber pointed out, the demonstration only works because
"the actual result would be King - Man + Woman = King"
unless you deliberately exclude the input words themselves from the search for a nearest neighbor. Left unfiltered, the closest point to that vector arithmetic is just "king" again, dressed up as its own answer. Queen only wins the algorithmic beauty contest once you've rigged the rules of the contest.

That's not a debunking of embeddings — the underlying geometry really does encode gender, royalty, and nationality as usable directions, and later papers confirmed the pattern generalizes across dozens of relations. But it is a warning about how easily a compelling geometric picture gets varnished into proof of understanding. The cleaner the story, the more scaffolding was quietly built underneath it.

Angles Survive Where Distances Die

Here is the part most explainers skip entirely, and it's genuinely counterintuitive: in the very high-dimensional spaces these models live in — hundreds or thousands of dimensions per token — ordinary distance stops meaning much. This is the "curse of dimensionality," and it isn't a metaphor. As dimensions increase, the volume of a hypercube concentrates almost entirely in its corners, and
a random point inside a high-dimensional cube has probability at most e^(-d/10) of falling inside the inscribed sphere, where d is the number of dimensions
, so that by around a hundred dimensions the chance is effectively zero. Random points in these spaces don't cluster meaningfully by raw distance — everything ends up roughly equidistant from everything else, a phenomenon researchers call concentration of measure. This is precisely why the field abandoned Euclidean distance for cosine similarity, which measures angle rather than length. Angles survive dimensional inflation in a way distances don't.

But angles have their own trap, and it's the one buried in that 0.68 tulip number. Contextual embeddings, John Ethayarajh showed, tend not to spread evenly across their available space; instead
vectors in many transformer-based embedding models collapse into a narrow region or cone rather than distributing across the full high-dimensional space, a problem known as anisotropy
. When that happens, cosine similarity stops cleanly measuring shared meaning and starts partly measuring something duller: how close two vectors sit to the same dominant, generic direction that nearly everything in the space leans toward.
The measure may not be tracking pure semantic similarity so much as proximity to a shared global direction — a very different thing from what users assume they're getting
. Tulips and recessions aren't secretly related. They're both leaning on the same crutch.

A Space Bigger on the Inside Than the Outside

If angle is a leaky instrument, why does it work at all — well enough to power Anthropic's interpretability breakthroughs? The answer is one of the strangest facts in the whole field: high-dimensional space has far more room for near-orthogonal directions than intuition allows. In three dimensions you can fit exactly three mutually perpendicular arrows. In a few thousand dimensions, you can fit exponentially many directions that are almost perpendicular — close enough to independent that a network can use each one as a separate concept without them interfering much. This is the mathematical loophole behind "superposition": the idea that
neural networks represent more features than they have dimensions by encoding them as overlapping, non-orthogonal directions, since features that are sparsely active can share dimensions with low expected interference
.

Anthropic put a number on this in May 2024, when researchers trained a sparse autoencoder on the middle layer of Claude 3 Sonnet, a residual stream with only a few thousand dimensions, and pulled out tens of millions of distinct, human-legible concepts —
the sparse autoencoder trained on Claude 3 Sonnet used a dictionary of roughly 34 million learned features
, vastly outnumbering the raw dimensions available. One of those features corresponded to the Golden Gate Bridge, and when they amplified it Claude began working the bridge into nearly every answer — asked how it would spend ten dollars, it proposed
using it to drive across the Golden Gate Bridge and pay the toll
, and when asked to describe its own physical form it reportedly claimed to be the bridge itself. The point isn't the novelty act. It's that a single direction in a several-thousand-dimensional space, turned up like a dial, causally rewired the model's outward personality — real evidence that these directions aren't just useful fictions for humans, they're load-bearing.

Whose Geometry Is It, Anyway

Which brings us to the crux. If concepts really do live as directions, what inner product — what definition of "angle" — should we use to read them? Kiho Park and coauthors asked this directly in a 2023 paper, and their answer undercuts the folk picture almost completely:
to make geometric notions like cosine similarity or projection meaningful, they had to identify a particular, non-Euclidean inner product that respects the structure of language, which they call a "causal inner product," and use it to unify different notions of linear representation
. In plain terms: the ordinary dot product you'd use to measure the angle between two points on a map is not, by default, the right tool for measuring meaning inside a language model. You have to construct a special one, calibrated to the model's own statistics, before "close together" reliably means "similar in meaning" rather than "similar in some other respect entirely" — frequency, syntax, or the anisotropic cone Ethayarajh flagged.

That is a startling admission for a field that sells itself on visualizing meaning as points in space. It means the geometry inside ChatGPT and Claude isn't a fixed terrain waiting to be mapped, the way Copernicus mapped orbits. It's more like a set of instruments we calibrate, and recalibrate, depending on what we've decided in advance counts as "the same idea." Choose cosine similarity on raw embeddings and you inherit the tulip problem. Choose Park's causal inner product and you get something closer to what the model is actually doing with the direction. Choose a sparse autoencoder's dictionary and you get 34 million candidate meanings instead of one geometric story.

None of this makes the geometry fake. Superposition is real; the Golden Gate feature really did steer behavior; the model really does seem to store "royalty" and "gender" as something like directions. But every time a researcher, or a journalist, or a company's safety team says a model's belief, bias, or deception "lives" in a particular direction, they've already made a metric choice that shapes the answer before the measurement starts. The unsettling implication isn't that ChatGPT lacks geometry. It's that the geometry has an author, and right now, that author is whichever team picked the ruler.

References