The way AI support usually gets deployed is that someone switches it on, it works for the three questions they thought to try, and the fourth customer finds the gap.
Helpie inverts that. The widget arrives switched off, and there is a proper place to work on it first.
The tester
An admin-only chat screen running the exact same assistant your customers will get. Same retrieval, same rules, same grounding enforcement. Not a preview or a simulation.
The difference is what it shows you alongside each answer:
- Which sources it retrieved, with titles, URLs and the score each one earned.
- What it did with your question before searching, including any alternative phrasings it tried when the first search came back empty.
- Whether grounding passed, and if not, why it was rejected.
- Tokens in and out, and the cost, shown to enough decimal places to be meaningful at Haiku prices.
Caching is deliberately bypassed here, so every test is a real answer rather than a replay of an earlier one.
What you actually do with it
Ask it the twenty questions your customers really ask. The ones in your inbox, not the ones on your FAQ page.
When an answer is wrong, the retrieval panel usually tells you why in one glance. Either it found the wrong page, which is a content problem you can see, or it found nothing, which means the answer is not published anywhere and now you know.
This turns calibration into editing your site rather than fiddling with settings. The fix for a bad answer is nearly always a better page.
Cost, before you commit
Because the tester reports real cost per answer, you leave calibration knowing roughly what a month looks like rather than guessing.
Test messages use your key and count towards your spending caps, which is deliberate. The number you see during calibration is a real number.
Publishing
When you are satisfied, switch the widget on. The tester keeps working afterwards, which is where you go when a customer reports a strange answer and you want to reproduce it without experimenting on the live site.
Why off by default
An assistant that says something wrong to a customer costs more than an assistant that arrives a week later.
Shipping it switched off is a small decision that changes the failure mode: the worst case becomes "we never got round to turning it on" rather than "it told a customer we do free returns."