We've all had the experience: you land on a site with a specific question, a little window pops up asking "Hi! How can I help you?", you type the question, and it answers something completely unrelated. You look for the close button and leave.
That chatbot didn't save a support ticket: it lost a customer. And yet the dashboard is going to show "1 conversation handled."
The problem isn't the model
With today's models, understanding the question is almost never the issue. The problem is design, and it shows up before a single line gets written.
It shows up in the wrong place. A chat that pops open on its own after three seconds interrupts someone who was reading. What it achieves is teaching people to close it on reflex, even when they need it.
It doesn't know what it doesn't know. A bot that confidently answers anything is worse than one that says "I don't know this, let me get you someone." Trust is lost once, and it doesn't come back.
It has no exit. If there's no clear way to reach a person, the chat stops being help and becomes a wall. And people notice immediately.
What actually works
That it knows what it's talking about. A generic bot connected to nothing answers generalities. One connected to your catalog, your stock, and your policies answers useful things: "do you have this in size M?" is the real question, and answering it takes knowing the stock, not being eloquent.
That it hands off quickly. The rule we use: if it can't solve it in two back-and-forths, it goes to a person with the full context of the conversation. Without making anyone repeat themselves, which is what annoys people most.
That it shows up when called. A visible button the user opens when they want to works better than a window that jumps out on its own. Fewer conversations, but almost all of them useful.
That it's obvious what it is. Pretending to be a person fools no one and destroys trust the moment it's found out. Saying "I'm an automated assistant, tell me if you need a person" sets the right expectation from the start.
What to measure (and what not to)
The metric almost everyone looks at is the number of conversations. It's useless: it goes up when the bot annoys people more.
The ones that matter:
- Resolution without a human. Of the queries it handled, how many ended without needing a person. And careful: "ended" isn't "the user left."
- Queries that stopped arriving. If the bot works, the support team gets fewer repeated questions. That drop is the real return.
- Abandonment. How many people open the chat and leave without typing. If it's high, the problem is the first impression.
- What people ask that it can't answer. This is the most valuable one and the least watched. It's a free list of what your site is failing to explain.
That last one usually ends up improving the site more than the bot.
When the answer is flat-out no
There are cases where it's not worth it:
- If your volume is low. With five queries a day, a person handles them better and it's cheaper.
- If your queries are all different. The bot shines at repetition. If every case is unique, there's no pattern to learn.
- If the error is expensive. In healthcare, legal matters, or money, a wrong answer costs far more than the query saved.
- If your information is outdated. A bot answering from stale data multiplies the problem instead of solving it.
The point
A good chatbot doesn't aim to answer everything. It aims to answer well what already gets asked a hundred times a week, and step aside quickly when it isn't its topic.
It's less ambitious and works a lot better.
In our AI case studies we show projects built with this approach. If you're evaluating one, get in touch and tell us what questions you're getting — we'll see if it qualifies.
