Sounding right isn't being right

"AI is unpredictable, unreliable and not ready for serious business." I think the people who say this have it backwards, and I'm betting big on it. But they're circling something real, and it's worth naming exactly: AI is very good at sounding right. Getting it to be right is a different job, and most of that job isn't about the model.

AI breaks the software contract

Traditional software is binary: it works or it doesn't. You build it, test it and deploy it, and it behaves exactly as specified, every time. Decades of enterprise technology rest on that predictability.

AI breaks the contract. It works in probabilities, not certainties. It doesn't execute instructions; it makes judgements. That brings a degree of randomness business technology has never had. The output is usually right. Sometimes it isn't. And the line between the two isn't always visible until something goes wrong and someone loses money or gets sued.

Most companies still think of AI as software. That's the wrong mental model, and it's where many early implementations failed. A probabilistic system needs different testing, different controls, different ownership and, in time, different law and governance.

The failure that looks like success

The obvious risk is the hallucination: the invented citation, the wrong number. Those are annoying, but they're catchable. The danger I worry about more is what I call alike reasoning.

Alike reasoning is output that carries every surface marker of logical thinking (coherent structure, the right terminology, plausible cause and effect, confident conclusions) and is still fundamentally flawed. To a non-expert it's nearly indistinguishable from real analysis.

We ran into it on an AI discovery project for an asset management company. Their client service department wanted faster responses and return projections generated from historical records and recent analyst research. The outputs were remarkably plausible: well-structured analyses, the right financial vocabulary, sensible-looking market correlations, confident projections.

Then our team sat down with their analysts to validate the reasoning. We found correlations read as causation, market conditions left out, and projections that ignored specific portfolio constraints.

Errors appeared in fewer than one in ten outputs. That sounds like a good result until you see what it does to the workflow. Because the errors were expertly camouflaged, every paragraph needed the same scrutiny as writing it from scratch. The hours saved on drafting went straight into verification, and verification was harder work, because the analysts couldn't skim. They had to read forensically.

That wasn't augmentation. It was beautifully formatted busywork. We were glad to learn it during discovery, before anyone committed months to building the wrong thing.

Data isn't the new oil

So what moves a system from sounding right to being right? Two things: an expert at the decision, and better inputs. The first is a matter of discipline. The second is where the real competition is.

Data is not the new oil. High-quality data is. The difference is bigger than it sounds.

Every era has its resource. Gas powered the factories, electricity rewired cities, oil made global trade possible. Then the economy moved to a different layer, and data became the resource. Google understood this before almost anyone. Its business was never really search; it was harvesting behavioural signals at scale and turning them into the most precisely targeted advertising machine ever built. Google, Meta and Amazon were, at their core, resource extraction businesses.

That era is over, and most organisations haven't registered it. Collecting more data stopped being an advantage once everyone did it. What matters now is narrower and harder: data that is high-quality, structured, current, consistent and well governed.

Feed that to a capable model with serious compute and the error rate drops. Not marginally. Meaningfully. That's what the asset manager's project was missing: fewer camouflaged errors means less forensic reading, and the time saved on drafting finally stays saved.

Data compounds

This is where it stops being a technical point. Fewer errors build trust with the people using the system. Trust drives adoption. Adoption creates feedback loops that make the system better. Better data, better models, better decisions, more capital to reinvest, and the flywheel spins faster for whoever got on first.

That's why AI isn't levelling the playing field; it's tilting it. The largest enterprises, with oceans of proprietary data, aren't just adopting AI. They're building competitive engines with it: custom models trained on internal data, decision pipelines that shave basis points off procurement and weeks off hiring. Mid-sized companies are bolting off-the-shelf LLM wrappers onto existing workflows. Fair enough; it helps. But when every competitor has the same ChatGPT, Claude or Gemini integration, it stops being a differentiator. Smaller firms are often still deciding whether the subscription is worth it.

It doesn't have to end that way. Amazon built warehouses while corner shops built free web pages, and then the internet produced Shopify, Stripe and a million niche businesses that couldn't have existed earlier. AI tooling is getting cheaper fast, and open-source models are closing the gap. But tooling is the part that gets cheaper. The proprietary data that makes your system meaningfully better than the generic one doesn't. That's the part most mid-market strategies ignore.

What the winners do

The winners won't necessarily be the biggest spenders. They'll be the ones who treat AI as a probabilistic colleague whose work needs checking, and as an architecture decision rather than a feature checkbox:

• Design for the error rate, not the demo. Decide up front what a wrong answer costs and who catches it.
• Measure the whole loop. Time saved in generation means nothing if verification eats it.
• Keep an expert at the decision, especially where money, compliance or people are involved.
• Clean the pipelines before there's a model to feed. The data work is slow and unglamorous, and it's the actual moat.
• Govern the data so people can trust it. Volume without governance is noise, not signal.
• Be honest about where off-the-shelf is good enough, and where custom work earns its keep.

The model you chose won't be your advantage. The data you feed it, and the people who check what it says, will be.