Number Class
← Back to Explore
Computer Science Artificial Intelligence & Data Vector Embedding Design

The 1,536 Number Was Never About Meaning

OpenAI's own newer model beats ada-002 using one-sixth the dimensions, exposing how much of that vector was dead weight

S.J. Nam 7 min read
The 1,536 Number Was Never About Meaning

Open any embedding vector from OpenAI's text-embedding-ada-002 model and look at position 196. It doesn't matter if you embedded the word "queen," a sonnet, or a grocery list — dimension 196 dips down in a strange, consistent spike, present in "every single vector embedding generated from the model - short ones, long ones, English ones, Spanish ones." Nobody at OpenAI has ever explained what that dimension is doing or why it behaves that way. It's just there, a fingerprint of an unexplained architecture, hiding inside 1,536 numbers that supposedly represent the meaning of whatever you typed.

That anomaly is the tell. The number 1,536 is treated across the industry as if it were a discovery — as if researchers had found the precise dimensionality that human meaning requires. It isn't. It's an inheritance, a compromise, and increasingly, evidence of its own excess. The real story of "why 1,536" is less about the geometry of meaning and more about the momentum of an industry that scaled first and asked questions later.

A Number Nobody Will Explain

Ask OpenAI directly why ada-002 outputs exactly 1,536 dimensions and you'll get silence — the company has never published a rationale. Developers have been asking in public forums for years. On GitHub, one confused engineer put it bluntly:
"Common model usually have 1024, 2048,.. dimentions output, why do you pick 1536?"
The honest answer is genealogical, not principled. When OpenAI cut ada-002 loose in December 2022, it replaced five older, specialized models, and its defining feature was that it was smaller than what came before:
"The new embeddings have only 1536 dimensions, one-eighth the size of davinci-001 embeddings, making the new embeddings more cost effective in working with vector databases."
Do the arithmetic and davinci-001's embeddings come out to 12,288 dimensions — which is not a coincidence, because that number is the hidden-layer width of GPT-3's largest configuration. Embedding size, in other words, was never derived from a theory of semantics. It was derived from whatever width the underlying transformer happened to have lying around.

Data scientist Vicki Boykis, who has tracked embedding sizes since the pre-transformer era, points out that the field's baseline used to be far more modest:
"A few years ago... 200-300 dimension embeddings were fairly common in industry, and that adding more dimensions during training would create diminishing returns."
She notes that OpenAI's jump to 1,536 dimensions arrived
"in line with GPT-3, which used much more training data than any previous model."
That's the whole genealogy: word2vec settled on 300 dimensions in the mid-2010s largely by convention, GPT-3 later needed a much wider hidden state to hold 175 billion parameters' worth of representation, and when OpenAI packaged a piece of that architecture as a commercial embedding product, the dimension count came along for the ride — first at 12,288, then, once cost became a concern, shrunk by a clean factor of eight to 1,536. No one ran a controlled experiment establishing that 1,536 is where semantic returns flatten out. It's what was left over.

The Sequel That Made Its Parent Look Bloated

Here is the detail that should embarrass the "bigger is better" instinct that got us to 1,536 in the first place. In January 2024, OpenAI shipped text-embedding-3-large and text-embedding-3-small, trained with a technique called Matryoshka Representation Learning (MRL), which — like the nesting Russian dolls it's named for — packs coarse, high-level information into the early dimensions of a vector and finer detail into the later ones. Ordinary embeddings treat every dimension as equally important; MRL deliberately does not, arranging things so that
"earlier dimensions store more information than dimensions later on in the vector, which simply adds more details."
The practical result stunned even OpenAI's own engineers: on the standard MTEB benchmark,
"a text-embedding-3-large embedding can be shortened to a size of 256 while still outperforming an unshortened text-embedding-ada-002 embedding with a size of 1536."

Sit with that for a second. A vector one-sixth the length of ada-002's supposedly essential 1,536 coordinates beats it outright. If 1,536 numbers were genuinely the minimum scaffolding required to hold a sentence's meaning, this shouldn't be possible — you can't remove five-sixths of a load-bearing structure and have it perform better. What actually happened is that ada-002's 1,536 dimensions were never load-bearing to begin with; a large fraction of them were redundant weight, present because the architecture produced them, not because meaning demanded them. MRL didn't discover a smarter way to compress information — it exposed how much of the original number was padding.

When the Angles Stop Meaning Anything

The counterargument here is obvious: surely more dimensions still help somewhere, otherwise why does text-embedding-3-large go up to 3,072? True — for capturing rare, fine-grained distinctions, extra dimensions retain value, and OpenAI's benchmarks bear that out. But there's a subtler cost hiding in high-dimensional space that has nothing to do with how much semantic content you can pack in and everything to do with what happens once you try to compare vectors at scale. As dimensionality climbs, distances between points tend to become statistically less distinguishable from one another — the so-called curse of dimensionality, where, as one review of vector search puts it,
"in high dimensions, even minor differences across many dimensions accumulate, inflating overall distances."
Cosine similarity was adopted precisely to dodge this, by measuring angle rather than raw distance. But even that fix has started to misbehave in the newest, most compressed models. Practitioners moving from older embedding tools to OpenAI's have noticed something odd: scores that used to spread across a full 0-to-1 range now huddle in a tight band, and
"OpenAI's modern embeddings seem to compress most results into a much narrower band around 0.2."
That has left developers
"wondering if these newer, more expensive models are actually worse at determining semantic similarity."
Nobody in the industry has produced a clean consensus answer. The geometry got more crowded, the tools for reading it got more indirect, and the interpretability got worse, not better, even as OpenAI insists performance keeps climbing.

What 1,536 Actually Tells You

None of this means embeddings don't work — they demonstrably do, powering everything from semantic search to recommendation engines. But the 1,536 figure specifically deserves to be read the way you'd read a hand-me-down suit that happens to fit reasonably well: it wasn't tailored to the problem, it was tailored to a different problem (training the world's biggest language model in 2020) and then handed down because throwing it out felt wasteful. The Matryoshka result is the quiet confession the industry has already made about this: most of those 1,536 numbers were never doing much work, and OpenAI's own newer models prove it by discarding five-sixths of them and winning anyway.

The interesting question isn't why your question becomes a 1,536-dimensional point. It's why so few people asked whether it needed to.

References