BLOG · 2026-05-30
The AI made it up (and it sounded convincing)
You ask the AI something and it answers with total confidence, with data, dates and names. It sounds flawless. The problem is that, sometimes, it just made all of it up. It doesn't lie out of malice — it works in a way that leads it to fill gaps with things that sound good even when they're false. This is called a "hallucination."
Why AI makes things up
A text AI isn't an encyclopedia that looks up the correct answer. It's, more accurately, a machine that's very good at predicting which word sounds right after the previous one. Most of the time that produces correct answers, because the correct thing usually also sounds right. But when it doesn't know something, it doesn't stay quiet: it fills the gap with what would sound true.
It's like a smart student in an exam: faced with a question they don't know, instead of leaving it blank, they write something convincing hoping to get it past the marker. The AI does that constantly, and does it well.
Why it slips through so easily
The dangerous part isn't that it makes mistakes — we all do. The dangerous part is the tone. The AI says false things with the same confidence as true ones. There's no "I think" or "I'm not sure." That's why we let our guard down: if it sounds so confident, it must be true. Well, not necessarily.
Where this hurts in a business
- A customer-service chatbot that invents a returns policy that doesn't exist.
- An automated summary that changes a key figure and nobody checks.
- A "legal" or "medical" answer that sounds professional but is made up.
- Quotes, laws or references that look real and don't exist.
The harm isn't the AI being wrong once. It's that, if you leave it alone, it's wrong at scale and with authority.
How to work with AI without getting fooled
- A person at the wheel for what matters. For anything with consequences (money, health, legal, decisions), have a person review before it goes out. The AI proposes, the person approves.
- Give it the data, don't ask it for the data. If you need it to use your prices or policies, give them to it in the moment, rather than trusting it to know them. Far more reliable.
- Make it cite its source. Asking where it got something helps catch fabrications: if it can't point to a source, be suspicious.
- Narrow its job. An AI that only answers about your specific information makes up far less than one you ask about everything.
Our stance: AI is a copilot, not an oracle
We use AI in our tools — for example in our security scan — but with one clear rule: the AI offers a quick, useful opinion, not the last word. Where there are consequences, there's a check behind it. So when we build AI automations, we do it so the AI helps a person decide better and faster, not so it decides alone and blind. AI is a brilliant copilot. A poor boss.