Every week I talk to a founder who sat through a slick AI presentation and learned how to spot a scripted AI demo, felt the hair on their arms stand up, and signed something. Six weeks later the thing does about a fifth of what they watched. The trick is not knowing more about AI than the person selling it. It is knowing which single request breaks a rehearsed performance and leaves a real system standing.
I build these systems for a living, so I know exactly how easy the theatre is to stage. I also know what an honest demo looks like from the inside, because mine have failed live in front of clients and I have had to explain why.
This post gives you the tells, the questions that break a fake, and the contract terms that mean you never have to trust a demo again.
Key Takeaways
- To spot a scripted AI demo, ask the vendor to run it on your data, live, with an input nobody in the room has seen before.
- A demo that only works in one order, with one phrasing, on one example record is a recording with extra steps, not a system.
- Ask what happens when the AI gets it wrong. Anyone with a real build answers in seconds, because they have already handled it.
- Latency tells the truth. Instant, perfectly formatted output usually means the answer was written before the meeting started.
- Insist on seeing the admin view: logs, error states, the queue of failed jobs. Polished front ends hide unfinished back ends.
- A small paid pilot on your own data beats any demo. Your systems, your records, a defined success measure, a fixed date.
- Put the demo in the contract. If it was shown during the sale, name it as a deliverable with an acceptance test.
Why the Scripted AI Demo Exists in the First Place
I want to be fair here, because not every scripted demo is a con. There are three different things going on, and they deserve different responses from you.
The first is the honest mock-up. Someone has built a visual prototype to show you what the finished thing would feel like, and they say so out loud. That is legitimate product design. Nobody has been deceived. If a vendor opens with “this is a clickable mock-up, the back end is not built yet, here is what building it involves,” you are dealing with an adult.
The second is the demo environment. The system is genuinely built, but it runs on a sample dataset that the vendor controls. Every AI company on earth does this, because you cannot hand a stranger live access to another client’s records. The demo is real software doing real work, just not on your business. This is normal and fine, and it becomes a problem only when the vendor lets you believe you are watching your future rather than their sandbox.
The third is the one that costs you money. A sequence of screens, prompts, and pre-generated outputs arranged to look like an autonomous system, presented as if it were finished, sold as if it were proven. The demo works because it has been run forty times and every input has been tuned until the output is beautiful. Change one word, and it falls over, which is why the salesperson keeps their hand on the keyboard.
Why this is so easy to fake right now
Two years ago, faking an AI demo took real effort. Today a competent person can produce a convincing one in an afternoon. Modern language models write plausible output on demand, so the vendor pre-generates the perfect response, saves it, and triggers it on cue. Front-end frameworks make a fake dashboard look production-grade in hours. Voice AI can be recorded, cleaned up, and played back as if it were live.
The cost of looking capable has collapsed. The cost of being capable has not. That gap is exactly where founders lose money, and it is why the old signals you used to judge a supplier- the polish of the deck and the confidence of the presenter- now tell you close to nothing.
The MIT number everyone quotes, and what it actually means
You have probably seen the statistic that around 95% of enterprise AI initiatives return nothing. It gets used as a stick to beat AI with. I read it differently. The projects that fail almost always started with technology rather than a process, and they were almost always bought off a demonstration rather than a diagnosis. Nobody in those rooms was stupid. They were sold a picture, and the picture was not load-bearing.
Nine Ways to Spot a Scripted AI Demo in Real Time
You do not need technical knowledge for any of these. You need to watch what is happening in the room rather than what is happening on the screen.
1. The input is always their input
Watch what gets typed. If every example is a lead named Sarah from a company called Acme, and every question is asked in the same phrasing, you are watching a rehearsed run. Real systems get shown with messy inputs because messy inputs are the actual job.
2. The output arrives too fast and looks too good
Language models take a few seconds to think. Voice AI has a small but audible gap before it answers. Data lookups take time. If a complex, formatted, perfectly structured answer appears the instant a button is clicked, either the answer was cached before the meeting, or the model was never called at all. Latency is one of the few things that is very hard to fake in the wrong direction.
3. Nobody else is allowed to drive
Ask to hold the laptop. Ask to type. In a real system, the answer is “sure”. In a scripted one, you will get a reason: the environment is fragile, the test account is shared, they will send you a login later. That hesitation is the whole answer.
4. It only works in one order
Scripted demos are linear. Step one, step two, step three, applause. Ask to skip to step three, or to go back and change step one, and the presenter will steer you back to the sequence. Real software does not care what order you use it in.
5. The failure case never appears
Across an entire hour, nothing goes wrong. No error, no timeout, no confused response, no edge case. That is not a sign of quality; it is a sign of curation. I have never run an honest live demo where something did not need a second attempt, and I say so upfront now because it saves everyone a strange moment.
6. Every screen is polished, and none of them is boring
Real systems are mostly boring. Settings pages, logs, configuration screens, a queue of jobs, a list of records. If every single screen you are shown is a hero screen, you are being shown a brochure. Ask for the ugly parts.
7. The integrations are described rather than demonstrated
Listen for the shift in tense. “It connects to your CRM” and “it will sync with your calendar” are claims. “Here is it writing to this CRM record, and here is the record” is proof. Integrations are where AI projects actually die, and they are the thing scripted demos skip most often.
8. Numbers appear without a source
A dashboard showing a 63% improvement is trivial to fabricate, because it is just text on a screen. Ask where the number comes from, which client, which period, and whether you can speak to them. Vague attribution is a tell on its own.
9. The system never says “I do not know”
Any AI worth buying has boundaries, and the people who built it know exactly where those boundaries are. If the demo suggests the thing handles everything, the vendor either has not run it in production or is not being straight with you. I would rather buy from someone who tells me the three things their system refuses to do.
The Questions That Break a Scripted Demo
Tells are useful, but questions are faster. These are the ones I would ask if I were on the buying side of the table, and roughly what a good answer sounds like.
“Can I give you an input right now?”
This is the single most useful sentence in the meeting. Bring one of your own records with you: a real enquiry from last week, a real customer question, a real quote request with the awkward details left in. Ask them to run it through, live, while you watch.
A real build handles it, possibly imperfectly, and the vendor talks you through what it did and why. A scripted demo produces a reason why not: the sandbox has different fields, your data has not been loaded, the model needs training on your business first. Some of those reasons are legitimate in isolation. All of them together, in response to one simple request, is your answer.
“What happens when it gets this wrong?”
Ask it plainly. Anyone who has run an AI system in production for more than a month has a fast, specific answer, because they have been woken up by this problem. They will tell you about the confidence threshold, the human review step, the escalation rule, the notification that fires. They will probably tell you a story about the time it went sideways.
A vendor who has not shipped will answer with reassurance instead of mechanism. “It is very accurate” is not an answer to “what happens when it is wrong”.
“Show me the logs”
Every real system keeps a record of what it did. Ask to see the last fifty things it did, including the ones that failed. This request is almost impossible to fake on the spot, and the reaction to it tells you more than the logs themselves. In a genuine build, this is a slightly boring screen full of timestamps and status codes, and the vendor will happily scroll through it.
“Who else has this running, and can I call them?”
Not a testimonial on a website. A person, at a business, with a phone. The answer does not have to be a household name. It has to be reachable. When I am asked this, I give a name, because a client who will take the call is worth more than any slide.
“What did you build, and what did you configure?”
There is a real difference between a company that has built a system and one that has assembled a demo out of off-the-shelf parts. Neither is wrong. Assembling proven parts is often the smarter, faster, cheaper choice, and I do it constantly rather than rebuild what already works. What matters is that they tell you which is which, because it changes who fixes it when it breaks and how quickly.
“What is the smallest version of this that you can put into my business next month?”
This question separates the sellers from the builders more cleanly than anything else. A builder will immediately start scoping something narrow and concrete, because that is how the work actually gets done. A seller will resist, because a small first step exposes the gap between the demo and the delivery.
What an Honest AI Demo Actually Looks Like
I want to give you the positive version too, because “be suspicious of everything” is useless advice. Here is what the good ones have in common.
It starts with your process, not their product
The best AI conversations I have been in started with the vendor asking questions for forty minutes before showing anything. What happens when a lead comes in. Who touches it. What breaks. Where the delay is. What the founder personally handles that nobody else can.
That is not stalling; it is the actual work. Automation only pays when it is pointed at a specific, repetitive, expensive process. If a vendor shows you a product before they understand a process, the fit is a guess, and you are funding the guess. This is the same reason I run a diagnostic before proposing anything, and why the fixes that stick are the ones aimed at a named bottleneck rather than a general sense that “we should be using AI”.
It is boring in at least one section
Somewhere in an honest demo, someone shows you a configuration screen and explains a rule. Or a review queue. Or how a human approves a message before it sends. These sections are dull, and they are the ones that prove the system exists, because nobody builds a fake settings page.
It admits its edges out loud
Voice AI is a good example. I have deployed a lot of it, and it is genuinely strong at answering, qualifying, and booking. It is weaker with heavy accents on a bad line, and it is the wrong tool for a complicated complaint from an angry long-term customer. Any honest demo of voice AI names those limits before you find them. The same is true of AI that writes: it drafts well, it needs review on anything sensitive, and the review step is a feature rather than an embarrassment.
The proof points are specific and traceable
Real results come with real texture. When I talk about database reactivation, I talk about James, a finance broker whose team had written off 319 dormant contacts. A multi-touch reactivation campaign, SMS and email, recovered $49,000 from a list he already owned and had already paid to build. That is a number attached to a person, a channel, a list size, and a mechanism you could go and inspect.
Compare that to “clients typically see a 40% uplift”. The second one is unfalsifiable, which is precisely why it gets used.
The vendor is comfortable being told no
Watch what happens when you push back or say you are not ready. Someone with a real business and real delivery capacity will say Fine, here is what to think about; come back when it matters. Someone who needs your signature this quarter will find a reason the price changes on Friday. Urgency that appears at the exact moment your enthusiasm dips is a sales technique, not a market condition.
Before You Sign: Turn the Demo Into a Deliverable
Even if you do all of the above and get good answers, do not rely on your judgment of a meeting. Put the protection in the paperwork, where it costs you nothing and changes everything.
Name the demo in the contract
The single most useful clause you can add: “the functionality demonstrated on [date] forms part of the deliverable.” One sentence. It reframes the entire demo from marketing into specification, and you will find out quickly whether the vendor believes their own presentation. Honest suppliers sign this without blinking, because they showed you something that exists.
Define what “working” means before work starts
Vague acceptance criteria are how projects rot. Write down what success looks like in a measurable form: every enquiry gets a first response within X minutes, the system books appointments into the live calendar without a human touching them, the reactivation campaign contacts every dormant record on the list and routes replies to a named inbox.
Then attach a date and a test. If the criteria cannot be written down, the scope is not understood yet, and an unclear scope is the most expensive thing you can buy.
Buy a small paid pilot instead of a big promise
I would rather a client pay for a narrow first piece of work on their own data than commit to a full build off a demo. A pilot is not free, and it should not be, because free pilots get deprioritised the moment a paying client sneezes. But it should be small, bounded, and pointed at one process with a measurable outcome.
The pilot answers every question the demo could not. Does this vendor communicate when things slip? Do they handle your actual data, with its actual mess? Does the thing survive contact with your team? Thirty days of that is worth more than any reference call.
Ask who owns what at the end
Where does the data live? Who owns the account. If you part ways in a year, what do you keep and what disappears? This matters more with AI than with older software, because the value accumulates in the context and history the system builds up about your business. If all of that lives in a platform you cannot export, you have rented the outcome rather than built an asset.
Check that they will still be there in eighteen months
Plenty of AI suppliers are one clever person and a weekend project. That can work, and some of the best builders are exactly that. But ask directly: how long have you been operating, how many clients do you support, who answers the phone when you are on holiday? The failure mode is not usually fraud. It is a solo operator who takes on more than they can support and quietly stops replying.
The Real Question Underneath All of This
Here is the thing I would want a founder to take away, and it is bigger than demo-spotting.
If you are evaluating AI vendors by how impressive their demo is, you are answering the wrong question. The right question is not “can this software do clever things?” Almost all of it can do clever things now. The right question is “which specific, expensive, repetitive part of my business is this pointed at, and how will I know it worked?”
Founders who ask that question rarely get burned, because it forces the conversation onto ground where fakery does not survive. It also tends to reveal that the most valuable automation is not the flashy one. It is the boring one that handles the thing you personally do forty times a week and have never told anyone else how to do.
A demo is a story about the future. A pilot on your own data is a fact about the present. Buy facts.
Conclusion
Scripted AI demos work because they are designed to bypass judgment and go straight to feeling. The defence is not scepticism about AI, which will cost you the genuine gains sitting in your business right now. If you want to spot a scripted AI demo, the defence is a handful of specific requests: give me the keyboard, run my input, show me the logs, tell me what happens when it is wrong, put the demo in the contract, start with a small paid pilot.
Behind all of it sits a bigger shift. The businesses pulling ahead are not the ones buying the most impressive tools. They are the ones building an AI brain that holds their context and an AI workforce that does the repeated work, one process at a time, with proof at each step. That is slower to demo and considerably harder to fake, which is exactly the point.
Ready to Test Your Own Situation?
If you are weighing up an AI proposal right now and want a straight second opinion on whether it holds up, book a 30-minute discovery call. No pitch deck, no pre-generated dashboard. I will ask about your process, tell you where I think the real bottleneck is, and be honest if the thing you have been quoted is the right buy. Sometimes the answer is that you do not need a vendor at all yet.
Frequently Asked Questions
What is a scripted AI demo?
A scripted AI demo is a presentation arranged to look like a working, autonomous system when the underlying build is incomplete or absent. Inputs are pre-chosen, outputs are often pre-generated, and the sequence only works in one order. It sits somewhere between an honest prototype and a misrepresentation, and the difference is whether the vendor tells you which one you are watching.
How can I tell if an AI demo is live or pre-recorded?
Ask to type the input yourself, using a real record from your business that nobody has seen before. Watch the response time, because genuine AI takes a few seconds to think and returns slightly imperfect formatting. Then ask to change the order of steps or go back and edit an earlier input. Live systems handle that. Recordings do not.
Should I pay for an AI pilot before committing to a full build?
Yes, and expect to. Free pilots get deprioritised whenever a paying client needs something, so they drag on and prove nothing. A small paid pilot on one process, with your own data, a written success measure and a fixed end date, tells you more about the vendor and the technology than any number of demos. Keep the scope deliberately narrow.
What questions should I ask an AI vendor before signing?
Six that matter: can I give you a live input right now, what happens when it gets something wrong, can I see the logs and error queue, who else is running this and can I call them, what did you build versus configure, and what is the smallest version you could deploy in my business next month. The reactions matter as much as the answers.
Is a very polished AI demo automatically a bad sign?
No. Plenty of genuinely good products demo well, and unpolished does not mean honest. The signal is not polish; it is whether the polish survives contact with your input. Ask for one boring screen: settings, logs, a review queue, a list of failed jobs. Real systems have those, and vendors will show them. Presentations do not have them.
How long should an AI proof of concept take?
For a single, well-defined process, expect a few weeks rather than a few months. If a vendor needs a quarter before you see anything working on your data, the scope is too broad, or the build is happening for the first time on your budget. Push for a narrower first piece with something demonstrable inside 30 days.
What should be in an AI automation contract?
Name the demonstrated functionality as a deliverable with the date it was shown. Write measurable acceptance criteria and a test date. Set out who owns the data, the accounts and the configuration if you part ways. Cover support response times and who answers when the main contact is away. Most disputes come from scope that nobody wrote down.
About Octavius
Titus Mulquiney is the founder of Octavius AI, where he builds AI brains and AI workforces for founder-led businesses stuck running everything out of their own head. Twenty years in marketing, ex-Sony product manager, ex-GM Zeal NZ. Based in Auckland, working with operators across NZ, Australia, and the US. Connect on LinkedIn.