Why AI Is Still Confidently Wrong

After years of watching TV and films, I have absorbed unconscious patterns of often-repeated tropes. Rather than keep them to myself, I share predictions about characters and plot with anyone willing to listen. I am often wrong, sometimes spectacularly right, but always confident.

In other words, I behave just like a large language model (LLM).

Of course, real LLMs exhibit a depth of knowledge, speed of retrieval, fluency of recombination, and an inexhaustible ability to follow up that exceeds any human.

And yet, most of us have experienced a definitive response that is entirely false, a reference to a non-existent paper, or a financial summary that does not add up.

For business leaders embarking on AI automation, this is a critical consideration that hasn’t received enough airtime.

What causes generative AI errors?

Let’s start with the word that has come to describe these phenomena: “hallucinations”.

In humans, these sensory illusions are induced by altered states caused by drugs, stress, sleep deprivation, or illness.

However, when AI produces false outputs, there is no external factor or error state because the model is working exactly as designed.

Since AI platforms have no sense of what is true or false – only the language patterns embedded in their model – they must rely on humans (or other systems) to spot these deviations.

But if these errors (let’s call them what they are) are not defects, how and why do they occur?

The clearest account is OpenAI’s own paper, Why Language Models Hallucinate.

The problem arises from the same statistical behaviour that makes LLMs so fluent: an imperative toalways produce a plausible next token.

That means a right or wrong answer is simply the statistically most probable response to the prompt, irrespective of its objective merit.

How often does this happen?

Error rates vary widely depending on the task and the environment.

For pampered corporate AI platforms in so-called “grounded” settings, where the data is quality-controlled and fed directly to the model, Vectara’s HHEM benchmark places the best frontier models in the 3–14% error range.

Similarly, on narrow reasoning tasks, leading models now err less than 10% of the time.

However, Stanford HAI’s 2026 AI Index reports that AI agents fail roughly one in three structured attempts when conditions get more realistic. These problems only compound as more agents interactwith one another.

The very worst failures seem to be reserved for citation generation, with one study finding that 69% of ChatGPT’s proposed references were invented.¹

These error rates are all higher than those of the traditional software they replace, which may not reflect the “progress” leaders expect.²

Encouraging developments, but no silver bullet

Since generative AI errors are not conventional software defects, they cannot be “engineered out”. What else can be done?

For the first time, the recent Claude Opus 4.8 model evaluates its own logic before delivering an answer. If it detects a gap or risk in its own reasoning, it flags this to the user as uncertainty, which means saying “I don’t know” more often. The model still makes a similar number of errors but is less prone to confident falsehoods. This is a big step forward – although a bit clunky in practice.

I have already mentioned “grounding”, where priming and checking AI models against accurate, timely, and well-structured corporate data can significantly reduce errors.

It is also possible to use generative models to check other models. A different LLM can review the first model’s output, challenge its claims, evaluate the evidence, and flag weak reasoning. This works because AI models tend to fail in different ways.

In some circumstances, putting traditional software back in the loop can help. AI-generated outputs are then subject to additional rules, workflows, permissions, and validation checks before becoming real-world decisions or actions.

Finally, and most importantly, humans need to be in the validation chain whenever possible to apply professional judgement and test AI outputs. After all, we ultimately own the output and actions of our AI – errors and all.

What should business leaders do?

AI errors are here to stay, and I recommend the following actions to manage their risk:

1. Recognise that errors are not a minor issue to be delegated to the AI team. They are board-level risks because they affect reputation, compliance, customer trust, financial reporting, legal exposure, and operational resilience.

2. Extend corporate governance to explicitly include AI. Legacy approaches do not cover all bases, so implement a specialised framework, such as the NIST AI RMF.

3. Don’t implement generative AI in situations that are critically sensitive to errors. LLMs are not suitable for high-accuracy tasks, particularly in quantitative settings. The prevailing assumption that generative AI will replace everything is, at the very least, premature.

4. At a strategy level, treat AI as a powerful but inherently unreliable form of corporate intelligence. We already know how to work with fallible human minds. AI should be brought under the purview of similar disciplines.

Conclusion

Generative AI is a breakthrough that is changing the way we do business – but we need to face up to its tendency to produce plausible but incorrect responses and actions.

The leadership task is not to eliminate errors – rather to build organisations that can benefit from fallible synthetic intelligence without being compromised by it.

At least Claude has finally mastered the art of admitting it doesn’t know everything. This will also be a vital quality for all business leaders seeking to avoid confident mistakes.

If you'd like to listen to an audio version of this article, please click here.

Footnotes

1. Damien Charlotin has logged over 1,400 court cases in which AI-fabricated references have surfaced in real filings.

2. The SaaS industry assumption is 0.1-0.5 defects per 1,000 lines of code. Large SaaS systems typically ship with 500+ residual problems, but these are typically resolved quickly and permanently for primary workflows.

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.