Number Class
← Back to Explore
Computer Science Artificial Intelligence & Data Vector Embedding Design

Chunking Isn't the RAG Bottleneck Everyone Blames It For

Two 2024 studies found smarter text-splitting barely helps — while chunk size tuned to question type and post-hoc context injection do

S.J. Nam 7 min read
Chunking Isn't the RAG Bottleneck Everyone Blames It For

The Two Chunks That Never Met

In a study of clinical decision support tools — the kind of software a physician might consult before adjusting a patient's medication — researchers testing retrieval systems found something unsettling.
Fixed-length chunks can split concepts or add noise, reducing precision.
A dosage guideline sliced neatly at the 500-token mark from its own contraindication. Both fragments got embedded, both got stored, and when a doctor's question came in, the system dutifully retrieved one half of the answer while the other sat, unread, in an adjacent vector. Nobody designed this failure. It emerged from a decision so mundane that most AI teams treat it as an afterthought: how do you cut up a document before you search it?

That question — chunking — is the unglamorous plumbing beneath every retrieval-augmented generation (RAG) system, the technique that lets a language model "search" a knowledge base by embedding pieces of text and matching them to a query. And the conventional wisdom, repeated in nearly every explainer on the topic, is that smarter cutting solves the problem: chunk by meaning, not by character count, and precision follows. The evidence doesn't fully support that story. What it supports instead is a less comfortable thesis: the industry has spent enormous engineering effort chasing elegant segmentation algorithms when the two things that actually move the needle are duller and less marketable — matching chunk size to the type of question being asked, and giving each fragment some memory of the document it came from.

Why Cutting Is Unavoidable

Embedding models and the language models that consume their output both have limits — a maximum number of tokens they can digest at once, and a tendency to represent long, topically diverse passages as a blurry average that flattens their most useful details. Feed an entire 40-page manual into a single embedding and you get a vector that represents everything and nothing. So documents get chopped, each piece embedded separately, so that a query about warranty terms doesn't have to compete with paragraphs about installation. This is the mechanical justification for chunking, and it's not in dispute. What is in dispute is what to do next.

The Cutting Contest Nobody Needed to Win

The default pitch for years has been "semantic chunking": use an algorithm, often another model, to find natural topic boundaries — paragraph breaks, sentence-similarity dips — rather than slicing every 512 tokens regardless of what's happening in the prose. It sounds obviously better, and it is obviously more expensive to compute at ingestion time.

A 2024 study bluntly titled "Is Semantic Chunking Worth the Computational Cost?" tested this assumption directly. The answer was a qualified no.
Semantic chunking occasionally improved performance, particularly on stitched datasets with high topic diversity.
Outside those narrow conditions, the benefits
did not consistently justify the additional computational cost
. In other words: the fancier method wins exactly when documents are artificially heterogeneous — stitched-together corpora built for benchmarks — and shrugs everywhere else. That's a genuinely counterintuitive result for a technique that gets marketed as a near-universal upgrade.

Size Is the Variable Everyone Underrates

If elaborate segmentation isn't the lever, what is? A 2025 multi-dataset analysis, "Rethinking Chunk Size for Long-Document Retrieval," found that
smaller chunks (64-128 tokens) are optimal for datasets with concise, fact-based answers
, while longer, more discursive question types performed better with larger windows. This is almost the opposite of a one-size-fits-all recommendation — it says the correct chunk size is a property of the question, not the document. A support system fielding "what's your return window?" wants tight, fact-dense fragments. A system answering "how does this policy interact with state law?" wants room for an argument to unfold across sentences. Teams that pick one chunk size for an entire corpus, as most do at launch, are optimizing for an average question that no user actually asks.

Two Labs, Same Diagnosis, Different Fixes

The deeper problem beneath both findings is that chunking destroys context by design. A fragment that says "it dissolves within 30 minutes" is useless if the reader can't see which drug "it" refers to — but the sentence naming the drug may sit three paragraphs earlier, in a different chunk. In September 2024, two independent lines of research converged on the same diagnosis from different angles. Anthropic proposed what it called Contextual Retrieval:
a strategy where an LLM adds additional context to each chunk
before it gets embedded, effectively writing a short reminder of what document and section the fragment belongs to. Around the same time, a separate research paper introduced "late chunking," a technique that embeds an entire document first using a long-context model and only afterward pools the token-level embeddings into chunk representations, so each chunk's vector already carries a trace of the whole text.
Late chunking requires no additional training and is applicable to a wide range of embedding models.

Neither approach touches where the cut happens. Both accept that the boundary problem is unsolvable at the level of segmentation and instead patch it after the fact by re-injecting context. That's a tacit admission that the industry's early obsession with finding the "right" place to slice a document was solving the wrong layer of the problem.

Where the Money Actually Gets Lost

None of this is academic.
Chunking artefacts arise when an answer is split across two chunks, so neither chunk contains the complete answer
— a failure mode engineers now see often enough to have a name for it. Meanwhile, a Gartner-sourced estimate cited in industry writing on RAG deployment notes that
only 48% of AI projects are successfully deployed into production
, with the successful ones taking months longer than planned. Chunking decisions, made early and rarely revisited, are a quiet contributor to that gap: teams pick a chunk size and a splitting library in week one, ship, and then spend months debugging retrieval quality without realizing the segmentation itself is the bottleneck.

The corrective advice that's emerging from practitioners closer to production is refreshingly unglamorous.
Start with recursive splitting, measure where retrieval fails using a proper evaluation framework, and add complexity only where failure modes justify it.
That's the opposite of reaching for semantic chunking because a blog post said it was more sophisticated. It's an argument for treating chunk size and structure as a hypothesis to be tested against real failures, not a design choice to be finalized in advance.

The Uncomfortable Lesson

Chunking will not disappear as context windows grow, because the problem was never really about how much text a model can hold — it's about how much text a query needs, and no model can guess that in advance. The teams building retrieval systems that actually work are not the ones with the cleverest segmentation algorithm. They're the ones willing to admit that a fragment, however cleverly cut, is still a fragment — and that the fix lies less in where you make the cut than in how honestly you account for everything the cut leaves out.

References