Number Class
← Back to Explore
Computer Science Artificial Intelligence & Data Natural Language Processing

AI Search Engines Cite Sources That Don't Say What They Claim

Across 1,600 tests, chatbots got citations wrong more than 60% of the time — one model failed 94%.

S.J. Nam 7 min read
AI Search Engines Cite Sources That Don't Say What They Claim

Type a question into an AI search engine and something strange happens before you even notice it: the machine has already decided, in a fraction of a second, what the internet says — and it has decided this independently of what the internet actually says. In 2024, Google's AI Overview told users that women who had abortions faced dramatically elevated risks of infection and hemorrhage, citing a real medical study as its source. The study existed. The numbers were even close to right. But
the referenced study found these risks in women who had opted out of abortions
— the opposite population. The citation was real. The link worked. The conclusion was inverted. That gap between "sourced" and "true" is not a bug hiding in the corner of AI search. It is the product.

Four Steps, One Weak Link

Ask an AI search tool a question and, mechanically, four things happen in quick succession. First, the system rewrites your query — expanding "best hiking boots for wide feet" into something closer to a research plan. Second, it retrieves: either pulling from a pre-built index of embedded documents or firing off a live web search, then ranking the results by relevance. Third, it chunks the winning pages into passages short enough to fit in the model's working memory and feeds them to a large language model, which drafts an answer conditioned on that material — this is the "retrieval-augmented generation" everyone in the industry name-drops. Fourth, and this is the step almost no one outside the labs talks about: the system attaches citations to the generated sentences, often as a separate process from the writing itself, matching claims to sources after the prose already exists.

That fourth step is where the abortion example, and thousands like it, come from. The citation-matching pass is a plausibility exercise, not a fact-check. It asks "which retrieved passage sounds most like this sentence," not "does this passage actually say this." A model can write a fluent, wrong summary and then bolt a real, correctly formatted, entirely mismatched footnote onto it. The result looks more rigorous than a traditional search results page — numbered references, clickable links, publication names — while being, in a meaningful number of cases, less trustworthy.

The Number the Marketing Never Mentions

Researchers at Columbia's Tow Center for Digital Journalism decided to measure exactly how often this goes wrong, and their numbers are the least-discussed, most damning data point in this whole conversation. In a study called "AI Search Has a Citation Problem," Klaudia Jażwińska and Aisvarya Chandrasekar
conducted 200 tests on eight different AI search engines: ChatGPT Search, Perplexity, Perplexity Pro, Gemini, DeepSeek Search, Grok-2 Search, Grok-3 Search, and Copilot
, feeding each one a direct quote from a news article and asking it to identify the headline, publisher, date, and URL. Across
the 1600 test queries, the search engines failed to retrieve the correct information more than 60% of the time
. The best performer was, fittingly, the tool that markets itself hardest on rigor:
Perplexity, which brands itself as a tool for research, had the lowest failure rate, answering incorrectly 37% of the time
. The worst was Grok-3, with what one summary of the follow-up findings called an
astonishingly high 94% error rate
— a newer, presumably more capable model performing dramatically worse than its predecessor at the one task, citation, that search engines exist to do.

This wasn't a one-off finding. An earlier Tow Center test of ChatGPT Search alone, run before the multi-engine study, found the tool
"confidently wrong in 146 cases"
out of 200 attempts — a 73 percent failure rate delivered with no hedging at all. And across the board, the researchers noted something more troubling than any single error rate: the chatbots were
generally bad at declining to answer questions they couldn't answer accurately, offering incorrect or speculative answers instead
. Ask a librarian a question she can't answer and she says so. Ask an AI search engine, and it guesses, cites, and moves on.

Confidence Is the Feature

It would be comforting to assume citation quality tracks with topic sensitivity — that engines get careful when the stakes are health, money, or law. A study published in October 2025 tested Microsoft Copilot specifically on dietary supplement questions across its three response modes and found the opposite of comfort. Overall accuracy hovered around one-third correct in every mode, while
72.7% of the citations came from unverified sources such as blogs, sales websites, and social media
. Roughly one in ten responses contained a citation whose source, when checked, simply didn't say what the chatbot claimed it said — the technical definition of a hallucinated citation, dressed up in the clothing of a real one.

Put these two studies side by side and the pattern sharpens into a thesis: the visible apparatus of AI search — the little citation markers, the source names, the "according to" phrasing — functions less as evidence and more as a trust signal detached from the thing it's supposed to signal. Readers extend confidence to a claim because it looks footnoted, the same reflex a print reader has toward a peer-reviewed paper. But a citation generated by a language model matching statistical patterns after the fact carries none of the guarantees that made footnotes trustworthy in the first place. It's citation theater: all the choreography of scholarship, none of the audit trail.

Who Benefits From the Guess

None of this is accidental, and following the money makes the incentive plain. Publishers who signed licensing deals with AI companies, hoping for accurate treatment of their journalism in exchange for access, learned that
these content licensing deals did not guarantee accurate citation in chatbot responses
. Meanwhile the underlying business logic has shifted in a way that makes a wrong-but-confident answer more valuable to the platform than a hedged, accurate one:
while traditional search engines typically operate as an intermediary, guiding users to news websites and other quality content, generative search tools parse and repackage information themselves, cutting off traffic flow to original sources
. A search engine that sends you elsewhere to verify a claim is, by its own commercial metrics, failing. A search engine that answers fluently and keeps you on the page is succeeding — regardless of whether the answer happens to be true. Even industry insiders concede the current state is a floor, not a ceiling; the Time COO quoted in the Tow Center's coverage offered the closest thing to a corporate mantra on the subject:
"I have a line internally that I say every time somebody brings me anything about any one of these platforms—my response back is, 'Today is the worst that the product will ever
" be.

Maybe so. But "worst it will ever be" is a promise about the future, not a description of the footnote sitting in front of you right now. The next time an AI search engine answers your question with a tidy paragraph and three source links tucked underneath, treat those links the way you'd treat a stranger's very confident tone at a dinner party — pleasant, persuasive, occasionally correct, and no substitute for opening the door yourself and checking who's actually there.

References