Every AI vendor demo looks great. The questions below are designed to find the gap between the demo and what actually happens after you sign.
Every AI vendor demo looks great.
The agent speaks naturally. It handles the scenario cleanly. The integration slide shows logos you recognize. The pricing looks reasonable. The case studies are impressive.
Then you sign, deploy, and spend the next three months discovering everything the demo didn’t show you.
Controlled environments. Pre-loaded with clean data. Scripted scenarios. No edge cases, no real caller behavior, no system failures, no awkward pauses.
A demo proves the technology works under ideal conditions. It proves nothing about how the system performs under real ones. Those are different tests, and most vendor evaluations only run the first one.
The questions below are designed to find the gap.
What happens when a caller goes completely off-script?
Ask the vendor to show you this live — not in a prepared scenario. Say something a real caller might say that wasn’t anticipated. Ask about a service the agent wasn’t configured for. Interrupt mid-sentence. Change the subject.
How the agent responds to the unexpected tells you more than how it handles the expected. Does it route gracefully? Get confused and loop? Make something up? The answer reveals whether the system was designed for production or for demos.
What does a failed call look like, and where does it go?
Every system fails sometimes. The question is what happens when it does.
Does the call drop silently? Go to voicemail? Route to a human? Get logged somewhere? Ask the vendor to walk you through a failure scenario — not a hypothetical, but an actual example from a live deployment.
If they haven’t thought carefully about failure modes, the system wasn’t designed for production.
Who writes and maintains the prompts?
A voice agent’s behavior is driven primarily by its system prompt. Who wrote it? Who updates it when your business changes — new services, updated hours, seasonal offerings? Who monitors it for drift when the underlying model is updated by the provider?
“We set it up and it just runs” is a red flag. Prompts require ongoing maintenance. The answer should include a named owner and a defined update process. If neither exists, the system will degrade silently over time.
How exactly does it connect to our existing systems?
Get specific. Not “we integrate with most CRMs” — which CRM, which fields, does it read and write, what happens when the connection drops, how long does a failed sync take to get detected and fixed?
Vague integration claims are where most post-deployment problems originate. Get the technical details in writing before you commit. If the vendor can’t answer specifically, the integration isn’t as solid as the slide suggests.
What does the handoff to a human look like?
Ask to see the escalation flow demonstrated live. How does the caller get to a human? How fast? Does the human receive context from the agent before saying hello, or does the caller start the conversation over?
A vendor who hasn’t designed the escalation carefully hasn’t thought about the caller experience carefully. These tend to come as a package.
What does the first 90 days look like after launch?
The demo is not the deployment. Real callers will do unexpected things. Edge cases will appear. The prompt will need adjustments. Integrations will need tuning.
What does the vendor’s onboarding and support process cover specifically? Who do you contact when something breaks on a Friday evening? What’s the response time? Is post-launch support included or billed separately?
Can we speak with a current customer in a similar business?
Not a testimonial on the website. An actual conversation with an operator running the system in a comparable context — similar size, similar industry, similar use case.
Ask them what surprised them after launch. Ask what they’d do differently. Ask what the vendor is slow to fix. The answer to that last question will tell you a lot.
Every question above tests the same thing: whether the vendor has thought seriously about what happens after the demo.
Good vendors have specific answers. They’ve dealt with edge cases before. They have a defined support process. They can point to real deployments that look like yours and tell you what the failure modes were.
Vendors who deflect, generalize, or get defensive at direct questions are showing you something. Believe it.
Build an evaluation scorecard before you talk to any vendor. Use the questions above as the criteria. Score every vendor the same way using the same questions.
The vendor who scores best on the hard questions — not the demo quality — is the one worth buying from. The demo is what they want to show you. The questions are what you need to know.
Hiring, collaboration, architecture review, or just a thoughtful systems conversation. No pitch deck required.