AI hallucinations: the near-strike on fabricated intel

September 20, 2026 · 5 min · Free, with full transcript

Can a language model's confident output survive contact with a high-stakes decision chain? This episode works through a case where a US military unit nearly authorized a strike on an intelligence packet containing AI-generated specifications for Chinese nuclear components that do not appear to exist. The position taken is blunt: hallucination is the model working as designed, and the real failure was upstream, in a pipeline that let unsourced claims sit in the same font and folder as verified human analysis. You walk away able to argue against three common beliefs, explain provenance tagging, recognise automation bias in review work, and apply one change that costs only friction.

Listen to the podcast

4 min

Chapters

Key takeaways

  • Tag every machine-generated claim so it is visually distinct from verified human sourcing, using colour, a banner or a header.
  • Treat fluency and accuracy as separate axes: models hedge on low-stakes questions and assert on high-stakes ones.
  • Do not read lower benchmark hallucination rates as safety; rarer errors are harder to catch and scale into thousands of fabrications across large document volumes.
  • Discount vendor-reported hallucination figures, including Google's, until there is independent audit.
  • A human in the loop only helps if that person has time, authority and provenance data; otherwise the review is theatre.
Read the full transcript

Host:Leaders' insights. Today, one hallucinated component list almost started a U.S. military strike. The mistake almost everyone makes is treating a hallucination like a bug you can patch out. It isn't. It's the machine doing exactly what it was built to do.

Expert:Right. A large language model, software that predicts the next plausible word, not the next true one, doesn't know the difference between a real Chinese nuclear component and a convincing sounding fake. It never did. Fluency and accuracy are separate axes and

Host:we keep confusing them. So walk me through what actually happened. A military unit nearly

Expert:ordered a strike off a fabricated component list? The intelligence packet included specifications for hardware that, as far as anyone can verify, does not exist. Part numbers, tolerances, the works, all generated, all wrong, all sitting inside a document that looked identical to human sourced analysis. Nobody flagged it because the output carried no marker saying

Host:I made this up. That's the part that chills me. It doesn't

Expert:hedge. It hedges when it's least dangerous and asserts when it's most dangerous. Ask it your dinner plans. You get maybe consider. Ask it for a weapons spec. You get seven confident decimal places.

Host:Let me bring you three things people believe about this. First one, hallucinations are getting rarer as models get bigger. So this problem solves itself.

Expert:Half true and the half that's false is the one that'll hurt you. Google's own numbers suggest their newer models hallucinate less on benchmark questions, though flag that Google is an AI lab selling these systems. And we have no independent audit to cross check them today. Even taking the figure at face value, rarer means the errors are harder to catch, not fewer at scale. A mistake that shows up one time in 500, spread across a million intelligence documents, is 2000 fabrications you now trust more because they're surrounded by correct ones.

Host:So bigger models make the survivors more camouflaged.

Expert:Precisely. You've bred a better liar, not an honest one.

Host:Some believe this was a human failure, not an AI one. Someone should have checked.

Expert:Wrong, and I'll die on this hill. The system was designed so that checking was optional. When you drop machine output into the same format, same font, same folder, as verified human work, you've engineered the shortcut. Blaming the analyst is like installing a door with no lock and blaming the guy who walked through it. The failure is upstream. In whoever decided AI-generated text needed no provenance tag.

Host:No, this came from a model-y stamp. Provenance meaning where the information

Expert:came from. The paper trail. Who or what produced each claim? In finance, a trade you can't source gets flagged automatically. In this pipeline, a claim you couldn't source sailed straight toward a launch authorization.

Host:Uncomfortable question, then. If provenance is that simple, why doesn't every serious

Expert:operation already do it? Because it's boring and it slows things down. And slow is the one thing high-stakes decision chains are punished for. Everybody wants the speed of the model and the accountability of the old process. You can't have both for free. You buy accountability with friction, and most organizations quietly decided friction was the enemy.

Host:Third belief, put a human in the loop and you're safe.

Expert:Half true. A human in the loop who's reviewing 40 documents an hour is not a safeguard. He's a rubber stamp with a pulse. The research on automation bias, the human tendency to defer to whatever the machine says, is brutal. People override a correct instinct because the confident output made them doubt themselves. So the loop only works if the human has the time, the authority, and the provenance data to actually push back.

Host:Strip any one of those, and the loop is theater. So what does someone running one of these

Expert:pipelines do Monday morning? Tag every machine-generated claim so it's visually distinct from verified human sourcing. Different color, a banner, anything. If a reader can't instantly tell what came from a model, your review process is fiction. That one change would have stopped this incident cold, and it costs you nothing but

Host:a little friction. Friction being the price of not launching

Expert:a missile at a rumor. Cheapest insurance in the building.

Host:Sources for today's episode, The Decoder TechCrunch AI, ours Technica AI, Google AI, vendor AI Lab. That's your five. The full AI track, structured and free of fluff, is at MBA-training.com.

Read and go deeper