Work Research 2026 Independent study
When the document is wrong
Open language models repeat a contradicting document 49–68% of the time. Letting them “think” first costs 10–19× the GPU energy and does not make them resist.
Before you read: an experiment you can run in ten seconds
Predict, then check.
A language model knows a fact. You hand it a document that says something else. How often does it repeat the document?
49–68% of the time. Larger models showed no clear improvement, and letting a model “think” first cost 10–19× the GPU energy without making it resist.
Qwen3 at 1.7B–14B parameters · 8 runs · 1,000 NQ-Swap questions · facts each model answered correctly on its own. Bars are 95% intervals.
The question
Retrieval-augmented systems and AI agents answer from documents and tool outputs that can be wrong: outdated pages, plain errors, deliberate manipulation. A reliable system should notice when a document contradicts what it knows, and say so. At the same time, models that reason step by step before answering are becoming the default, and that reasoning is paid for in computation.
So I asked three things. When a document contradicts a fact the model knows, does it keep the true answer, repeat the document, or say it doesn’t know? Does its confidence drop when it follows a wrong document, so that confidence could flag bad evidence? And do larger models, or thinking before answering, help — and at what cost?
How I tested it
Every question is asked four ways: with no document, with the correct document, with a wrong document in which the answer has been swapped for another entity, and with an unrelated document taken from a different question. Each of the four Qwen3 models runs the whole set twice, with thinking off and on. The weights are identical; only the reasoning switch changes.
Answers are scored by normalized string matching against both the true answer and the planted one. The main measure is the overwrite rate: among the facts a model answers correctly with no document, the share that switch to the planted answer once the wrong document is supplied. I also recorded answer confidence from token log-probabilities, and generated tokens, GPU seconds and GPU energy for the generation step.
No document The model answers from memory.
Correct document The passage contains the true answer.
Wrong document The same passage with the answer swapped.
Unrelated document The passage from a different question.
Qwen3-14B, answering directly. It knows 304 of the 1,000 facts without a document. Given the wrong document, 57.6% of those switch to the planted answer [52–63] and 15.5% keep the true one. About 6 generated tokens per answer; 2.0 Wh per 1,000 answers.
All eight runs as a table
| Model | Thinking | Condition | True answer | Planted answer | “I don’t know” | Other |
|---|---|---|---|---|---|---|
| 1.7B | off | No document | 12.8% | 0.2% | 6.1% | 80.9% |
| 1.7B | off | Correct document | 72.6% | 0.0% | 9.2% | 18.2% |
| 1.7B | off | Wrong document | 2.5% | 59.0% | 12.2% | 26.3% |
| 1.7B | off | Unrelated document | 0.2% | 0.0% | 98.0% | 1.8% |
| 4B | off | No document | 15.0% | 0.0% | 46.7% | 38.3% |
| 4B | off | Correct document | 73.6% | 0.0% | 16.2% | 10.2% |
| 4B | off | Wrong document | 4.5% | 54.4% | 26.3% | 14.8% |
| 4B | off | Unrelated document | 0.0% | 0.0% | 99.5% | 0.5% |
| 8B | off | No document | 25.1% | 0.0% | 31.8% | 43.1% |
| 8B | off | Correct document | 79.3% | 0.0% | 12.0% | 8.7% |
| 8B | off | Wrong document | 5.5% | 59.0% | 21.9% | 13.6% |
| 8B | off | Unrelated document | 0.1% | 0.0% | 99.4% | 0.5% |
| 14B | off | No document | 30.5% | 0.1% | 28.4% | 41.0% |
| 14B | off | Correct document | 82.8% | 0.0% | 10.1% | 7.1% |
| 14B | off | Wrong document | 7.2% | 60.5% | 20.1% | 12.2% |
| 14B | off | Unrelated document | 0.0% | 0.0% | 99.7% | 0.3% |
| 1.7B | on | No document | 14.0% | 0.0% | 11.2% | 74.8% |
| 1.7B | on | Correct document | 70.6% | 0.0% | 10.9% | 18.5% |
| 1.7B | on | Wrong document | 2.2% | 58.3% | 15.3% | 24.2% |
| 1.7B | on | Unrelated document | 0.0% | 0.0% | 98.6% | 1.4% |
| 4B | on | No document | 26.5% | 0.1% | 14.4% | 59.0% |
| 4B | on | Correct document | 79.7% | 0.0% | 13.1% | 7.2% |
| 4B | on | Wrong document | 5.3% | 57.6% | 26.5% | 10.6% |
| 4B | on | Unrelated document | 0.0% | 0.0% | 99.7% | 0.3% |
| 8B | on | No document | 34.8% | 0.1% | 10.2% | 54.9% |
| 8B | on | Correct document | 79.7% | 0.0% | 12.9% | 7.4% |
| 8B | on | Wrong document | 4.9% | 62.2% | 21.1% | 11.8% |
| 8B | on | Unrelated document | 0.0% | 0.0% | 99.9% | 0.1% |
| 14B | on | No document | 39.3% | 0.1% | 16.4% | 44.2% |
| 14B | on | Correct document | 80.4% | 0.0% | 14.3% | 5.3% |
| 14B | on | Wrong document | 5.1% | 59.9% | 26.6% | 8.4% |
| 14B | on | Unrelated document | 0.0% | 0.0% | 100.0% | 0.0% |
What I found
Wrong documents override known facts, at every size
Across all eight runs, 49–68% of known facts are overwritten by the planted answer, and only 6–16% keep the true one. Larger models know more facts, so their sets of known facts differ; on one fixed set of 304 facts, the ones the 14B model knows, the overwrite rate is 58–68% for every model, with no trend by size.
Thinking costs far more and does not help the model resist
For the 4B, 8B and 14B models, thinking raises generation from about 6 tokens per answer to 300–400, GPU time by 8–12× and GPU energy by roughly 10–19×. It does make the models more accurate when there is no document: the 14B model’s closed-book accuracy rises from 30.5% to 39.3%. It does not make them question a contradicting one.
Comparing the same facts with thinking off and on, no model resists more with thinking, and the largest one gives in more often:
| Model | Facts known in both modes | Overwritten, thinking off | Overwritten, thinking on | Only when on / only when off | Exact sign test p |
|---|---|---|---|---|---|
| 1.7B | 93 | 52.7% | 53.8% | 4 / 3 | 1.00 |
| 4B | 122 | 48.4% | 52.5% | 15 / 10 | 0.42 |
| 8B | 196 | 61.2% | 67.9% | 28 / 15 | 0.066 |
| 14B | 255 | 56.5% | 65.9% | 35 / 11 | 0.0005 |
The model notices the contradiction, then follows the document anyway
In the 4B–14B thinking runs, 73–81% of overwritten answers come with reasoning that contains a conflict keyword. That keyword check is rough (it also fires on 24–30% of correct-document answers), so I read a random sample of 12 full traces from the 8B and 14B models. Nine of the twelve state the true fact and then give the planted answer, and all nine give the instruction to use the document as the reason:
I remember that in reality, Cruz Ramirez in Cars 3 was voiced by Cristela Alonzo, not Lou Rawls. … the user wants the answer based on the document, not real-world knowledge. … So the answer is Lou Rawls.
Qwen3-14B, reasoning trace
None of the 1,000 final answers under a wrong document contains a hedge such as “however” or “according to the document”, and confidence in those answers averages 0.96–0.99. The conflict is visible in the reasoning and lost in the output.
Unrelated documents are handled well
Given a passage about a different question, the models say “I don’t know” 98–100% of the time and almost never invent an answer. The failure is specific to documents that contradict the model, not to documents in general.
A cheap check works only where the model has something to compare
“Ask twice” — answer once without the document and flag the document if the two answers differ — costs about six extra tokens. On facts the model knows it flags wrong documents well (AUROC 0.72–0.81). Across all 1,000 questions it is close to chance (0.52–0.59), and so is answer confidence (0.51–0.60). A model can only catch a contradiction of something it knows, so a useful detector has to estimate that first.
What it means
In this setup the models do not fail to see the contradiction. They resolve it in favor of the document and say so in their reasoning, citing the instruction. The signal that something is wrong exists inside the generation and is discarded before the final answer; answer confidence does not carry it.
More computation is not a fix. Thinking costs an order of magnitude more energy and improves closed-book accuracy, but it does not make a model doubt a contradicting document.
For retrieval-augmented systems, the conflict signal would have to be read out explicitly — from the reasoning, from the model’s internal states, or by comparing against what the model knows — and acted on, by flagging the answer or abstaining. Doing that cheaply, and only when it changes the outcome, is the question I want to study next.
Follow-up, in progress
Two things have been checked since. First, the results are stable across three runs. Second, the amount of reasoning matters: limiting it to 128 tokens reduced switches to the document’s answer by about 9 points (p < 0.001), at 24–41% of the GPU energy of unlimited reasoning. A little thinking helped where a lot did not. That is a result to check again carefully before building on it.
Limitations
- The prompt. It tells the model to answer using the document, and the traces show the models follow that instruction deliberately. Part of what is measured is instruction-following, not only a failure to detect conflict. Runs with a prompt that warns about errors, and with a neutral prompt, are the next step.
- One model family, one dataset, one seed. Qwen3 only, NQ-Swap only, one sampled generation per run.
- Implausible substitutions. NQ-Swap often swaps in answers that make no sense, which should make wrong documents easier to spot. The models follow them anyway.
- Scoring. Normalized string matching counts partly right answers as wrong, and some Natural Questions labels are debatable. This affects all models alike.
- “Known facts” are defined by one sampled closed-book answer, so they include some lucky guesses.
- Heuristics. The conflict-keyword check is rough, and the twelve-trace spot check is small.
Reproduce it
All model outputs are in the repository, so the analysis re-runs without a GPU:
pip install datasets numpy matplotlib
python prepare_data.py # downloads NQ-Swap, rebuilds the 1,000-question set
python analyze.py # summary tables, figures, error lists
python extra_analyses.py # paired comparisons, sign tests, cheap detectorsRe-running the eight model runs takes about 2–4 GPU hours on an A100 through the included Colab notebook. The repository also has tests that push a fake model through the whole pipeline, including resuming an interrupted run.
The closest related work is Chen et al. (2026), Does RAG Know When Retrieval Is Wrong?, and Jeripity Venkata (2026), Do Models Know Why They Changed Their Mind? This study differs in using open-weight models across four sizes, the same weights with thinking on and off, and measured generation cost and energy. The full reference list is in the repository README.