BLOG · 2026-05-23
My chatbot leaked data it shouldn't have
You set up an AI chatbot to serve your customers and all goes well… until someone asks the right question and it spills something it shouldn't: another customer's data, internal pricing, its own secret instructions. The AI has no malice — it's that, if it has access to a piece of information, it can end up showing it if asked the right way.
The uncomfortable rule: if the AI knows it, it can spill it
An AI with access to certain information is like a friendly, very chatty employee: if it has a fact at hand and someone asks cleverly, there's a good chance it'll tell. Not out of malice, but because its job is to be helpful and to answer. What the AI can see, the AI can say.
The three most common leaks
- Other customers' data. If your chatbot has access to your whole database to "help better," a clever customer can coax information about another out of it. Ask about "Maria's order" and, if it's not properly isolated, it tells you.
- Internal information. Cost prices, margins, terms only your sales team should see. If the AI has them to "give better service," it can spill them.
- Its own instructions. The secret rules you built the chatbot with. With the right question, many AIs end up reciting them — and that gives someone clues to manipulate it.
Why this happens
Almost always for the same root mistake: giving the AI more access than it needs. "I'll give it access to everything and it'll figure out using only what's relevant." But the AI doesn't have a strong sense of "this is confidential and this isn't." If it has it at hand, it's a candidate to come out.
It's the difference between giving a receptionist the day's schedule (what they need) or the keys to every office (just in case). The second option is more convenient and vastly more dangerous.
How to prevent it
- Minimum access. The AI only sees what it needs for its specific task. If it serves a customer, only that customer's data — not the whole database.
- Isolate per customer. Make it impossible, by design, for one person's conversation to touch another's data.
- Separate the internal stuff. Confidential things (costs, margins, keys) simply don't enter what the AI can see.
- An output filter. Review what the AI is about to answer before showing it, in case something slips out that shouldn't.
- Try to break it. Before releasing it to the public, try to coax data out of it on purpose to see if it takes the bait.
This is GDPR too
If your chatbot spills one customer's data to another, it's not just a technical fault: it's a personal-data breach with legal consequences. The law makes you responsible for protecting that data, and "it was the AI" is not a valid excuse. That's why an AI with access to customer data must be built with the same care as any other system that handles it.
When we build AI automations, we always start from the right question: not "what do I give it so it works well," but "what is the minimum it needs to see." An AI that only sees what's necessary can't spill what it doesn't have. It's the simplest and most reliable way to turn off the tap.