You can trust AI citations to show you where to look, but not to prove where an answer came from. A study posted to arXiv on November 12, 2026 measured the gap. In the AI systems it tested, 1.2% to 15.7% of citable claims came from the model's memory, not from the passages it had looked up.
The standard automatic check could not tell the difference, so treat a citation as a pointer, not a receipt. Before you repeat any number, date, price or recent fact from an AI answer, open the cited page and find the sentence that says it.
What are AI hallucinated citations?
An AI hallucinated citation is a source a chatbot such as ChatGPT gives to back up its answer, when that source doesn't exist or doesn't say what the answer says. OpenAI defines hallucinations as "plausible but false statements generated by language models". A bad citation is the same problem wearing a footnote.
The first two kinds below are familiar. The November 2026 study measured a third, quieter kind, where the source is real and on topic but the answer didn't come from it.
| Citation problem | What you see | Does clicking the link catch it? |
|---|---|---|
| Made-up source | A paper, author or web address that doesn't exist | Usually, because the link is dead or the paper can't be found |
| Real source, unsupported claim | A real page that never says what the answer says | Yes, if you read the part that matters |
| Cited but unused | A real, relevant page, while the claim itself came from the model's memory | Only if the page and the answer disagree and you read closely |
Here is the third kind in everyday terms. You ask an AI about a supplier's return policy, and it links the supplier's own policy page. The link is real, but the answer recites an older policy the model picked up in training, so the page and the answer quietly disagree.
Does the AI actually read the sources it cites?
Not always: in the Cited but Unused study, posted to arXiv in November 2026, 1.2% to 15.7% of citable claims in the AI systems tested came from the model's parametric memory, not from the passages it retrieved. Parametric memory is simply what the model absorbed during training.
The full title is Cited but Unused: Parametric Provenance in Retrieval-Augmented Generation, by Tanzeel Abbas, Tehseen Zia and David Windridge. It studies retrieval-augmented generation, or RAG, where the AI looks things up first, then answers and attaches citations to show where each claim came from.
The authors sum up the problem bluntly: "These citations frequently misstate provenance, and standard verification cannot detect this". Provenance just means where a claim really came from.
How the researchers caught answers coming from memory
The researchers swapped one fact in the retrieved passage for a false version and asked the model again. If the answer changed to match, the model was reading the passage. If it didn't follow the change, the claim came from memory.
Think of an open-book quiz where you secretly change a date in the student's textbook. A student who still writes the original date wasn't reading the page they're pointing at, however neatly they cite it.
What the numbers show
The tests covered five sizes of one model family, plus a second model family for comparison.
| Measure | Result | What it means for you |
|---|---|---|
| Share of citable claims that came from memory, all models tested | 1.2% to 15.7% | From just over one claim in a hundred to nearly one in six |
| Same share, most conservative version of the test | 4.6% and 8.5% (two models) | Even the cautious count is well above zero |
| Effect of model size | The rate rose as models got larger, within a family | Bigger did not mean safer |
| Standard automatic check (NLI) | Results indistinguishable from chance | The usual checker can't spot these claims |
Why the usual automatic check missed it
The standard automatic check is natural language inference (NLI), a method that asks whether one piece of text supports another. In the study, NLI gave results indistinguishable from chance across models, even when it looked only at the cited passage.
The authors say the cause is structural: "a model answering from memory selects a citation that agrees with its output". A support check asks does this page agree with the answer? The real question is did the answer come from this page?, and NLI only asks the first one.
The researchers also compared ten attribution methods (ways of tracing which source a claim came from) on the same claims. Acting on the best signal still did not make citations more honest at the rates they measured. In the authors' words, their best detector "measures what the model knew rather than what it used", and they call separating the two an open problem.
What the study did and did not test
- It is a preprint. The arXiv page lists no journal or conference, so treat it as early research, not peer-reviewed work.
- The models are unnamed. The abstract doesn't name the model families or datasets, so the numbers can't be pinned to any brand of chatbot.
- It tested research setups, not apps. The study ran RAG systems in a research setting and did not test any consumer chatbot directly, including the four covered below.
Why does ChatGPT make up sources, and does it still happen?
ChatGPT can still get things wrong: as of November 2026, OpenAI's own explainer on why language models hallucinate says accuracy "will never reach 100%". Part of the reason, OpenAI says, is that most evaluations (the tests used to score models) work in a way that "encourages guessing rather than honesty about uncertainty".
Three forces push an AI toward a confident answer with a shaky source.
Guessing gets rewarded
OpenAI says strategically guessing when uncertain improves accuracy but increases errors and hallucinations. In plain terms, a guess sometimes lands, but it also adds wrong answers. OpenAI's SimpleQA example shows the trade-off:
| SimpleQA result, per OpenAI | gpt-5-thinking-mini | OpenAI o4-mini |
|---|---|---|
| Abstention rate (gave no specific answer) | 52% | 1% |
| Error rate (wrong answers) | 26% | 75% |
The model more willing to say I don't know made far fewer errors.
A thin page makes it bolder
Google Research says models like Gemini, GPT and Claude answer well when given enough context, but cannot reliably avoid wrong answers when the context falls short. In its tests, Gemma gave wrong answers on 10.2% of questions with no context, but on 66.1% when given context that wasn't enough.
Picture asking about a bank's overdraft fee when the search only finds the bank's savings page: the model answers anyway. Google Research also found a paradox: looking things up generally makes answers better overall, yet it makes the model worse at holding back when it should.
It picks a source that agrees with it
This is the Cited but Unused finding. A model answering from memory chooses a citation that matches what it already said, so the link looks like support even when it played no part.
For real cases and the checks that catch them, see our explainer on why AI hallucinates.
Can you trust Perplexity, ChatGPT, Gemini and Claude citations?
As of November 2026, you can trust Perplexity, ChatGPT search, Gemini Apps and Claude API citations as pointers, not as proof. Each vendor's own docs describe links or pointers to sources, and none of the four publishes how often its cited claims really came from memory.
The Cited but Unused study did not test these products. So the table sets out what each vendor says in its help pages and docs, and where that promise stops.
| Tool | What the vendor says its sources do | What that does not prove | Best for |
|---|---|---|---|
| Perplexity | Numbered footnotes on each answer link to the original sources so you can verify (Perplexity's help center) | That the footnoted sentence was taken from that source | Research questions where you will open the footnotes |
| ChatGPT search | Links to sources such as news articles and blog posts, plus a Sources button that opens a sidebar of references (OpenAI) | That each sentence came from a listed page | Current topics with clear public pages to read |
| Gemini Apps | A Sources button at the bottom of the response or in-line, when sources are available (Google) | That each sentence came from a link; Google says links may include content related to parts of the response | Finding leads to read further |
| Claude API Citations | Citations are guaranteed to contain valid pointers to the documents you provide (Anthropic) | That the claim came from the passage pointed to; the guarantee covers the developer API, not the Claude.ai app | Questions about your own files, where you can open the passage |
Say you use Perplexity to compare two invoicing apps. The footnote next to supports recurring invoices is your shortcut to the vendor's page, not proof that the feature exists.
Gemini Apps won't always show sources: Google's help page says "Not all responses include related links or sources".
Anthropic's guarantee is about the pointer: its developer docs say citations are "guaranteed to contain valid pointers to the provided documents". Think of a library reference that always leads to a real shelf. Useful, but silent on whether the writer read the book.
Google's developer docs say grounding with Google Search lets Gemini give more accurate answers and cite verifiable sources beyond its knowledge cutoff. Grounding means letting Gemini search the web before it answers, and the knowledge cutoff is the date its training data ends.
That's a vendor claim. On our reading of the study, fresh pages are also where the risk is highest, because the page may say something newer than what the model remembers.
How do you check if an AI citation is real?
To check an AI citation, open the cited page and find the exact sentence that states the claim. As of November 2026, none of the automatic checks compared in the Cited but Unused study was accurate enough to tell whether a claim came from the page or from the model's memory. So this step is still yours.
Save that effort for the claims where the model's memory and the page are most likely to disagree.
The two-question test
This rule is our interpretation of the study, not one of its findings. A model answering from memory picks a citation that agrees with it. So the risky claims are the ones where the page probably says something different from what the model learned in training.
Before you trust a cited claim, ask yourself:
- Could this be different now than a year or two ago, or different for me than for most people?
- Is it an exact number, date, price, name or quote?
If either answer is yes, open the source and find the sentence. If both are no, the citation is a bonus, not a chore.
| Claim type | Everyday example | Risk if the answer came from memory | Your move |
|---|---|---|---|
| Recent news | Who won last week's vote | High | Find the sentence on the cited page |
| Changed facts | A software plan's current price or a new rule | High | Check the figure and the page's date |
| Local or niche facts | Your council's bin collection days | High | Prefer the official local page |
| Your own documents | Your team's travel policy in an uploaded file | High | Ask for a quote, then search the file |
| Exact numbers and quotes | A percentage in a market report | High | Search the page for the exact figure |
| Stable, widely known facts | What photosynthesis is | Low | A glance is enough unless it goes in a report |
| General explanations | How compound interest works | Low | Open the source only if you need more depth |
The quick page check
- Open the link and confirm it is the page the answer names: right site, right title, sensible date.
- Press Ctrl + F (Cmd + F on a Mac) and search for the key number, name or phrase.
- Read the sentence around it: same fact, same place, same time period?
- No match means the claim is unsourced, however confident the answer sounded.
For a longer routine with a practice passage, use our citation verification workflow for AI claims. If you cite research papers, our comparison of Elicit, Consensus and SciSpace shows how three research tools find real papers.
Make the AI hand you the quote
Anthropic's guide to reducing hallucinations suggests several habits, including these four:
- Give the model permission to admit uncertainty.
- Have it cite quotes and sources for each claim.
- Tell it to use only the provided documents, not its general knowledge.
- Have it retract any claim it can't back with a quote.
Anthropic says these techniques significantly reduce hallucinations but don't eliminate them entirely. It wrote that advice for Claude, but nothing stops you using the same wording in other chatbots:
Answer only from the sources you cite, not from your general knowledge.
After every number, date, price or name, quote the exact sentence from the source.
If you cannot find a supporting quote, say so and remove that claim.
It is fine to say you are not sure.
A quote can't prove where the answer came from. But it hands you the exact words to search for, and if they aren't on the page, you've caught the problem in seconds.
Do this next
Three habits catch the riskiest cases without turning every chat into homework:
- Read citations as pointers, not receipts. A real, relevant link tells you where to look, not where the answer came from.
- Run the two-question test on anything you will forward, publish or act on. If the claim is new, changed, local, internal or exact, open the page.
- Paste the quote-first instruction into research chats. Then search the page for each quote before it goes into an email, report or client deck.
If you only do one thing, make it this: search the cited page for the exact number before you repeat it.
