Hallucinations

An AI hallucination is a confident, incorrect answer: a language model states something fluent and plausible, yet it is false. In agentic engineering, hallucinations are the central failure mode. An agent that invents an API, a file path, a citation, or a passing test result will keep working as if the invention were real.

The clearest account of why this happens comes from a September 2025 paper by researchers at OpenAI and Georgia Tech. The thesis is as follows: language models hallucinate because the way they are trained and graded rewards guessing over admitting uncertainty. A confident guess scores well. "I don't know" scores nothing.

The Short Version

• Hallucinations begin in pretraining as ordinary statistical errors.
• They survive later training because almost every major benchmark scores a wrong guess the same as "I don't know."
• Guessing therefore always scores at least as well as honesty, so models learn to guess.
• The fix is not bigger models or cleaner data. It is changing how answers are graded.

Where Hallucinations Come From

In pretraining, a model learns to imitate the statistical patterns of its text. The paper frames a hallucination as a classification mistake: the model cannot reliably separate a true statement from a plausibly false one. Several consequences follow.

1. They are process errors. The same process that produces correct text produces convincing fakes. Asked about one of the paper's authors, DeepSeek-V3 gave three different wrong birthdays, and ChatGPT, DeepSeek, and Llama each invented a different dissertation title.
2. Generating is harder than checking. The authors define a yes/no task, "Is this output valid?" A model's error rate when generating an answer is at least about twice its error rate on that yes/no check. A model that cannot tell a fact from a convincing fake will produce fakes.
3. Clean data does not prevent them. Even with error-free training data, the standard pretraining objective still produces errors. Real data contains mistakes, so real error rates run higher.
4. Facts seen once get hallucinated. Some facts have no pattern to learn from, such as an individual's birthday. If 20% of birthday facts appear exactly once in the training data, the base model should get at least about 20% of birthdays wrong. The estimate builds on Alan Turing's "missing mass" work. Facts repeated often, like Einstein's birthday, rarely go wrong.
5. Some errors are built into the model. Asked how many D's are in "DEEPSEEK," models answered 2, 3, even 7, partly because they read tokens rather than letters. DeepSeek-R1 gets it right by spelling the word out, which points to the model's design rather than its data.
6. Other causes exist. Some problems are intractable for any computer, such as breaking encryption. Unusual prompts unlike the training data cause errors, and mistakes already in the training data get repeated.

One point matters for anyone who assumes a well-behaved model should never err: a calibrated base model has to be wrong some of the time. Calibration means the model's confidence matches how often it is right, and matching the uncertainty in the data forces some mistakes. GPT-4's base model was well calibrated before reinforcement learning.

Why Training Does Not Remove Them

If pretraining plants hallucinations, later stages could in principle prune them. They mostly do not, and the reason is how models are scored.

1. Models sit in exam mode. A student guessing on a multiple-choice test never scores worse than one who leaves it blank. The paper proves the same for models: when a correct answer scores 1 and both a wrong answer and "I don't know" score 0, "I don't know" is never the best choice.
2. Honest models lose on leaderboards. Consider two identical models where one says "I don't know" when unsure and the other always guesses. The guesser scores higher on accuracy-based benchmarks, so optimizing for leaderboards teaches models to bluff.
3. Nearly every major benchmark rewards guessing. The authors checked ten influential ones: GPQA, MMLU-Pro, IFEval, Omni-MATH, BBH, MATH, MuSR, SWE-bench, HLE, and WildBench. Nine give no credit for "I don't know." WildBench's 1–10 rubric can even score "I don't know" below a confident answer that contains hallucinations.
4. AI graders make it worse. Language models used as judges sometimes mark long, wrong answers as correct, even in math, which rewards bluffing further.
5. A separate hallucination benchmark is not enough. Its results get outweighed by the many mainstream benchmarks that reward guessing. A narrow habit, like opening every reply with "Certainly," is easy to fix with one targeted test because nothing else rewards it. Hallucination is not, because almost everything else does.

The Proposed Fix

The paper's remedy is a grading change, not a model change.

1. State the penalty in the prompt. Tell the model to answer only when its confidence exceeds a threshold t: a correct answer earns 1 point, "I don't know" earns 0, and a wrong answer costs t/(1−t) points. The penalty is chosen so the arithmetic itself settles the decision. If the model believes it is right with probability p, then guessing is worth p − (1−p)·t/(1−t), and that quantity is positive only when p is greater than t. Below the threshold, the expected loss from a wrong guess outweighs the point a lucky guess would earn, so staying silent is the better move; above it, answering pays. Making t larger makes the penalty steeper — at t = 0.75 a wrong answer costs 3 points, at t = 0.9 it costs 9 — so the model keeps quiet unless it is nearly certain.
2. Change the benchmarks that matter. The penalties belong in widely used benchmarks such as SWE-bench, not only in new side benchmarks. The authors call this a socio-technical fix, because the influential leaderboards are the ones that have to adopt it.
3. Measure behavioral calibration. Rather than asking a model to state a probability, check that it answers only when it is above the stated confidence, and audit that behavior by comparing accuracy and error rates across different values of t.

Two cautions close the argument. Search and reasoning do not cure hallucination on their own: when a search comes up empty, accuracy-only grading still rewards a guess, and retrieval cannot fix errors like letter counting. Also, calibration is not a hallucination measure. A model can post a strong calibration score without being trustworthy, so the score by itself is a poor guide to how often it hallucinates.

Why This Matters For Agentic Engineering

Agentic workflows raise the stakes because agents act on their own output. A hallucination that a person would catch while reading becomes an input to the next step when an agent runs unattended. The pattern from the research points to a few practical habits.

• Facts an agent states once, with no source, are the most likely to be invented.
• A fluent, confident answer carries no signal about whether it is correct.
• Verification steps such as tests, source checks, and a second-pass review, are where hallucinations get caught, not the phrasing of the answer.

Catching these failures reliably starts with reasoning from what is actually known rather than what merely sounds right.

September, 2026