Number Class
← Back to Explore
Computer Science Artificial Intelligence & Data Computer Vision

Face Search Works Because It Throws Your Face Away

FaceNet compresses a photo to 128 numbers, then a second shortcut algorithm skips most of the database anyway — a double approximation that hits some faces harder than others.

S.J. Nam 8 min read
Face Search Works Because It Throws Your Face Away

The Photo That Became 128 Numbers

In 2015, three researchers at Google took a photograph of a human face — every wrinkle, shadow, and asymmetry it contained — and crushed it down to 128 floating-point numbers. Nothing more. Their system, FaceNet,
achieves a new record accuracy of 99.63% on the widely used Labeled Faces in the Wild dataset
, and it did so by training on a firehose of data: roughly
200 million images spanning 8 million identities
. The headline number is impressive. The stranger fact is what made it possible: to get better at recognizing faces, the engineers had to stop looking at more information and start looking at radically less.

That inversion is the real story of image and face search, and it's the one most explainers skip in their rush to describe cosine similarity and vector databases. The dominant narrative treats face search as a triumph of scale — more data, more compute, more accuracy. The evidence suggests something less comfortable: every face-search system in production today, from Google Images to Clearview AI to the reverse-search tool PimEyes, works only because its designers made a series of deliberate, lossy compromises to escape a mathematical trap. And the shape of those compromises — not just biased training photos — is a direct cause of why the technology fails some faces more than others.

Why More Data Makes You Harder to Find

Here is the counterintuitive part. Intuition says that describing a face with more numbers — more pixels, more measurements, more dimensions — should make comparisons more precise. Geometry disagrees. Researchers studying nearest-neighbor search have shown that as the number of dimensions grows, distance itself stops discriminating: a query point's nearest neighbor and its farthest neighbor start to look almost equally far away. Beyer and colleagues formalized this by defining a query as
unstable if all of the points in the dataset are nearly the same distance from the query point
— at which point, as the same analysis notes,
the query can be thought meaningless since there is little reason to return any one point over another
. Raw pixel space, with its tens of thousands of dimensions, is exactly this kind of trap. A literal pixel-by-pixel comparison of two photos of the same person, taken in different light, is often less "similar" by the numbers than two photos of different people taken under matching studio conditions.

So the problem facing FaceNet's designers wasn't really "how do we compare faces." It was "how do we escape the geometry of high-dimensional space long enough for comparison to mean anything at all."

Triplet Loss: Cheating the Curse on Purpose

Their answer was to stop trying to preserve information and start training the network to discard it selectively. FaceNet's training procedure, built around what's called triplet loss, shows the network three images at a time — an anchor, a second photo of the same identity, and a photo of someone else — and nudges the network's weights until the same-identity pair sits closer together in the 128-dimensional space than the different-identity pair. Do this across millions of triplets and the space stops being a neutral coordinate system; it becomes a purpose-built terrain where "closeness" has been forced to mean "same person," overriding all the incidental closeness that lighting, angle, or camera noise would otherwise create. It's telling that when engineers tuned the size of this embedding, more dimensions didn't obviously help: work on high-accuracy face embeddings describes networks compressing features down to
a 128 dimension embedding for an RGB input image
, the same order of magnitude FaceNet settled on, trained via contrastive objectives so that
embeddings of perceptually similar images... are closer to each other
— with closeness defined, again, by nothing more exotic than a Euclidean or cosine measurement.

That last detail — contrastive, 128-dimensional embeddings — is not unique to face search at all. Apple used the identical architecture, right down to the 128-number output, for NeuralHash, its since-shelved system for spotting known illegal images without ever seeing the photo itself. Whether the target is a face or a piece of contraband imagery, the mathematical move is the same: replace an enormous, ambiguous space with a small, disciplined one where a simple distance threshold can stand in for a judgment about identity.

The Approximation Stacked on the Approximation

Compressing a face into 128 numbers solves the meaningfulness problem, but not the speed problem. A search engine holding hundreds of millions of faces cannot, for every query, measure the exact distance to every stored vector — the arithmetic would be trivial per comparison but ruinous in aggregate. So a second layer of approximation gets stacked on top of the first. Systems like Facebook's FAISS library implement structures such as Hierarchical Navigable Small World graphs, built on the method from
"Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs"
, in which every embedding becomes a node connected to a handful of neighbors, and a query greedily hops through the graph toward the closest cluster rather than checking every point in the database.

This means the "closest match" a face-search tool hands back isn't the mathematically closest vector in its entire archive — it's the best candidate a bounded, greedy graph traversal happened to surface. Speed was purchased with a second, quieter compromise on top of the embedding's own compromise. Two layers of approximation sit between the photo you upload and the "97% match" a screen displays, and neither layer was optimized for truth. Both were optimized for throughput.

Whose Errors Are Cheap

This is where the stakes stop being abstract. NIST's 2019 Face Recognition Vendor Test set out specifically to
determine whether, and to what degree, face recognition algorithms differed when they processed photographs of individuals from various demographics
. Subsequent analysis of that body of work reports that NIST found
increased false positive identifications in women, children, and the elderly, alongside higher false negatives in under-represented racial groups– particularly Black, East Asian, and Native American individuals
. Vendors dispute how much this matters in their own systems — Clearview AI, for instance, maintains that in NIST's one-to-one comparison tests
its algorithm consistently achieved greater than 99% accuracy across all demographics
. That disagreement is itself informative: the same federal testing program is being read by different parties as proof of a solved problem and as proof of a persistent one, and both readings can be defended by picking a different accuracy metric out of the same report.

But the usual explanation for demographic gaps — "biased training data, needs more diverse photos" — undersells the geometry. A shared 128-dimensional embedding space is, structurally, a single compromise applied to everyone at once. Whatever facial variation was statistically rarer in training gets mapped, on average, into a tighter, more crowded region of that space, simply because the network had fewer contrastive triplets to spread that variation apart. Tighter clustering means the same universal similarity threshold that cleanly separates well-represented faces will, in the crowded region, both misfire more (false matches) and fail more (missed matches). You cannot fix this by collecting a slightly more diverse photo set and retraining with the same architecture; the compression itself, by design, treats "faces that differ" as a global average rather than a guarantee for every subgroup.

The Bet Behind the Percentage

None of this makes face search useless — FaceNet's 99.63% is real, and HNSW graphs really do make searching a billion photos practical on ordinary hardware. But every confidence score these systems output is the residue of two stacked approximations built for speed, not certainty: a learned space that had to throw away information to become meaningful, and a search algorithm that had to stop looking early to stay fast. The number on the screen looks like a fact about the world. It is closer to a wager, placed inside a room whose walls were built, quite deliberately, too small to hold the whole truth — and the people standing in the most crowded corners of that room were always going to be the ones the bet gets wrong most often.

References