Learning how to vet AI automation references is critical before you hand a stranger the keys to your CRM, your phone line, and your customer database. And the only thing standing between you and a bad decision is a reference call you feel weird about making.
I have sat on both sides of this. I have been the provider being checked, and I have been the buyer who did not check and paid for it later. Most founders skip the step or do it badly, because asking another business owner “did this actually work” feels like an imposition.
It is not. Here is what to ask, what the answers really tell you, and how to run the whole thing in two short calls without either party feeling awkward.
Key Takeaways
- Vet AI automation references by asking about the messy parts of the build, not the outcome. The outcome is always good; that is why they were offered as a reference.
- Ask for two references: one from the last three months, one from over a year ago. The old one tells you what happens after the honeymoon.
- The single best question is “what broke, and how long did it take them to fix it?” Every real build breaks something. Only a fake one never did.
- A provider who cannot produce a reference in the same size band as your business is telling you something, even if they never say it out loud.
- Reference calls take 15 minutes. Open with what you are considering, ask five questions, and offer to return the favour. That removes the awkwardness.
- Ask the reference who actually did the work. If the person who sold it was not the person who built it, find out whether that mattered.
- No references at all is not automatically a no. No references plus no willingness to stage the payments is.
Why Founders Skip This Step, and What It Costs
Nearly every founder I speak to has a version of the same story. They bought something automated, it half worked, and now it sits there costing money and doing nothing useful. When I ask whether they spoke to another client before they signed, the answer is almost always no.
The reasons are consistent. It feels like extra work at the exact moment you have decided to move forward. It feels rude to ask a busy stranger for 15 minutes. And there is a quiet worry that asking too many questions makes you look difficult, or worse, makes the provider walk away.
That last one is backwards. Any provider who gets defensive about a reference request has just given you the most useful piece of information in the whole sales process, for free.
The cost of skipping it is not usually the money. It is the twelve months. A bad automation build does not fail loudly on day one. It works in the demo, limps through onboarding, then quietly degrades while everyone is too busy to look at it. By the time you accept it is not working, you have spent a year not fixing the actual problem, and you are now more sceptical about AI than you were before you started.
MIT research on enterprise AI projects found that around 95% deliver no measurable return, and the difference between the ones that work and the ones that do not is almost never the technology. It is the process around the build: who scoped it, who owns it, whether anyone changed how they work. A reference call is the cheapest way to find out whether your provider does that part, because their previous client lived through it.
There is a second cost that nobody talks about. When a build fails, the knowledge fails with it. Whatever the provider learned about your business goes out the door with them. You are back to running everything out of your own head, which is where you started, except now you are also out of pocket and a year older.
What You Are Actually Checking
Before you write a single question, get clear on what a reference call is for. It is not for confirming that the provider is good. The reference has already been selected to say that. You are checking four specific things, and only one of them is about results.
Did they scope honestly? The most common failure in this industry is not incompetence; it is overselling. A provider says yes to a scope they cannot deliver, and the client finds out in month three. The reference will tell you whether what was promised in the proposal matched what showed up.
What happened when it broke? Every automation build breaks something. A phone number routes wrong, a workflow fires twice, an integration drops a field. That is normal. What is not normal is a provider who goes quiet for a week. The response to a broken thing is the truest signal of what you are buying.
Who actually did the work? In a lot of shops, the person who charms you on the sales call is not the person who touches your system. Sometimes that is fine; sometimes the handover is where everything falls apart. The reference knows.
Are they still using it? This is the question that separates a real result from a nice launch. Plenty of builds go live successfully and get abandoned within six months because nobody adapted the team’s habits around them. Ask.
Notice that none of these is “were you happy”. Happy is not information. The four above are.
How to Ask for References Without the Awkwardness
The awkwardness comes from ambiguity. Neither side knows what the call is for, so both sides tiptoe. Remove the ambiguity and the discomfort goes with it.
Ask the provider properly
Do not say “do you have any references?” That invites a link to a testimonial page. Say this instead:
“Before I sign, I’d like to speak to two of your clients. One you finished in the last three months, and one from over a year ago. Fifteen minutes each. Can you set that up?”
The specificity does the work. You have named the format, the time commitment, and the type of client. A provider with real clients will say yes and have it organised within a couple of days. A provider without them will start negotiating the terms of your request, and that is your answer.
The one-year-old reference matters more than the recent one. Recent clients are still in the glow. They have a new system; everyone is excited; nothing has been tested by a busy month yet. The client from eighteen months ago has watched the thing survive or not survive contact with reality. That is the call you want.
Ask for the right size of business
If you are a ten-person business turning over a couple of million, a reference from a solo consultant tells you almost nothing. The problems are different, the stakeholders are different, the failure modes are different. Ask for a reference in your size band, and ideally with a similar shape of operation, meaning a similar volume of calls, leads, or jobs.
If they cannot produce one, that is not automatically disqualifying. It might mean you would be their first in your bracket, which is a legitimate thing to be, as long as everyone says so out loud and the commercial terms reflect it. What you cannot accept is a provider who dodges the question and hopes you forget.
Open the call so the other person relaxes
Reference calls are awkward mainly because the person on the other end does not know how honest they are allowed to be. Give them permission in the first thirty seconds:
“Thanks for taking the time. I’m looking at working with [provider] on [thing]. I’m not looking for a glowing review; I assume it went well, or you wouldn’t be on this call. What would actually help me is the honest version, including the parts that were annoying. I’ll do the same for you one day if you ever need it.”
That last sentence matters. It converts the call from you extracting value to two business owners trading it. Almost everyone loosens up after that, and the useful material starts coming out around minute four.
Keep it to 15 minutes and mean it. Watch the clock, say “I said fifteen and we’re at fourteen, one last question”, and then actually stop. People remember that, and you will get a better call for it.
The Questions That Actually Vet AI Automation References
Five questions, in this order. The order matters, because the early ones are easy and the hard ones land better once the conversation is warm.
1. “What did you buy, and what did you end up with?”
Open wide and let them talk. You are listening for drift between the two halves of that sentence. A little drift is healthy; it means the provider adapted as they learned the business. A lot of drift means the original scope was fiction. If they say “we bought a call answering system and ended up with a call answering system plus a follow-up sequence we didn’t know we needed”, that is a good sign. If they say “we bought a full build and ended up with about half of it”, write that down.
2. “What broke, and how long did it take to get fixed?”
This is the question. Ask it plainly and then be quiet.
If the answer is “nothing broke”, one of two things is true. Either the build was small enough that nothing could break, which tells you about the scale of what the provider actually delivers, or the client is being polite. Follow up gently: “Not even a small thing in the first month?” Something always comes up in the first month.
What you are listening for is the response time and the tone. “It broke on a Friday, I texted him, it was sorted by Saturday morning” is worth more than any case study. “It took about three weeks, and I had to chase twice” is a real answer about what month seven of your relationship will feel like.
3. “Who did the work, and did you always know who to call?”
Sales-to-delivery handover is where a lot of these engagements quietly fall over. You are checking whether there was one named person accountable throughout, or whether the client got passed down a chain. Ask whether the person who scoped it ever disappeared. Ask who they contact now when something needs changing, and how long it takes to get a reply.
4. “Are you still using it, and has anything changed since it went live?”
The best possible answer here is “yes, and we’ve added two more things since”. That tells you the first build worked well enough to earn more work, which is the only vote that costs the client something.
A worrying answer is “yes, it’s running” delivered flatly, with no examples of anything having changed in a year. Automation that never gets adjusted is usually automation nobody is looking at. Ask what they would change if they could. If they have a list and nobody has actioned it, you have learned what ongoing support looks like in practice.
5. “What would you do differently if you started again?”
This is the closer, and it almost always produces the most useful sentence of the call. It gives the reference a socially safe way to criticise the process without criticising the person. You will get answers like “I’d have got my team involved earlier”, “I’d have cleaned up the database first”, or “I’d have started with one thing instead of three”. Every one of those is a direct instruction for your own build.
If you have time for a sixth, ask what the provider was like to have around during a bad week. Everyone is pleasant in the sales phase. You want to know what happens when the client is stressed and something has gone wrong.
Reading the Answers and the Red Flags Worth Walking Away From
Answers are only half of it. How the provider behaves around the request tells you as much as anything the reference says.
Red flags from the provider
They offer written testimonials instead of a phone call. Written quotes are marketing assets. They are collected at the highest emotional point of the relationship and edited before publication. Useful as colour, useless as evidence. A provider with genuinely happy clients can produce a live human on a phone.
They insist on being on the call. Occasionally this is innocent; more often it is a chaperone. The whole value of a reference call is candour, and candour does not survive a third party. If they want to introduce by email and then step back, that is normal and fine. If they want to sit in, decline politely and see what happens.
Every reference is from the last eight weeks. Either they are new, which they should just say, or their older clients would not take the call. Both are worth knowing.
They get prickly. The provider’s emotional reaction to a routine due diligence request is a preview of their reaction to your first complaint. I would rather lose a deal to a client who checks properly than win one from someone who did not, because the second client is going to be the harder relationship anyway.
Red flags from the reference
Vagueness about numbers. A real client can usually tell you something concrete, even roughly. Bookings went up. The phone stopped going to voicemail after five. We recovered X from a list we’d written off. If everything is described in adjectives and nothing in numbers, the result may be softer than the case study suggests.
They cannot explain what the system does. If the reference cannot describe their own automation in two plain sentences, nobody handed it over properly. That is a training and documentation problem, and you will inherit it.
Enthusiasm about the person, silence about the outcome. “He’s a lovely guy, really responsive” with nothing after it is a soft no dressed up as a yes. Push once: “and commercially, did it do what you needed?” The pause before the answer will tell you.
Green flags most people miss
The best sign is a reference who complains about something small and specific, then immediately says they would do it again. That combination is very difficult to fake and almost always means the relationship is real. “The onboarding took longer than he said, mostly because our data was a mess, but honestly I’d hire him tomorrow” is the single most reassuring sentence you can hear on one of these calls.
The second best sign is a client who has bought more than once. Repeat purchase is the only endorsement with money behind it.
For a sense of what a genuine result looks like when it is written up properly, our AI automation case studies show the numbers and the timeframe rather than the adjectives. Dr Claire, a dental practice owner, was losing close to half her inbound calls with two receptionists on the desk. After we put an AI receptionist on the line, missed calls went to zero and booked appointments rose 44%. That is the shape of claim you should be able to verify on a reference call in about ninety seconds.
What To Do When There Are No References
Sometimes there genuinely are none. The provider is new, or they have signed confidentiality agreements, or their only comparable client is a direct competitor of yours and will not take the call. None of that makes them a bad choice. Some of the best builds I have seen came from people early in their own story who were hungry to get it right.
What it does mean is that you replace reference risk with structural risk management. Three things do that job.
Stage the money. Split the engagement into milestones with a payment attached to each one, and make the first milestone small and quick. Something that goes live within two weeks and that you can see working. If they will not stage it, that is the real red flag, not the missing references.
Make them show you their own system. This is my favourite test, and almost nobody uses it. Ask the provider to show you what they have built for themselves. Not a slide deck, the actual thing. If they sell automation and run their own business on memory and a shared inbox, you have learned something important. The people worth hiring built it for themselves first and will happily open the laptop.
Get the exit written down. Who owns the accounts, the phone numbers, the workflows and the data if you part ways in six months? Can you take it with you, or does it die with the relationship? This belongs in the agreement regardless of how many references you got. Our AI automation contract checklist covers the specific clauses worth insisting on.
If you are still at the stage of drawing up a shortlist, comparing AI automation companies in NZ is a reasonable place to start before you get to reference calls at all. And once you have signed with someone, set a 90-day checkpoint in the calendar now, while you are still clear-headed about what you expect.
The Bigger Question Behind All of This
If you are nervous enough about a purchase to want references, the nervousness is usually not really about the provider. It is about the fact that you cannot tell, from the inside, whether the thing you are buying is the right thing to buy. You know the phone gets missed. You know the follow-up is inconsistent. You know there is a database sitting there that nobody has touched. But you cannot see the whole shape of the problem clearly enough to know which piece to fix first, so you are relying on a salesperson to tell you.
That is the actual risk. Not a dodgy vendor, an unclear diagnosis. A well-built automation pointed at the wrong problem is still a waste of a year.
The fix is to get the diagnosis before you get the quote. Understand which tasks are eating your week, which of them a system could genuinely take over, and what order they should come off the list in. Then taking the time to vet AI automation references becomes a much simpler exercise, because you are no longer asking “is this person good”; you are asking “has this person done the specific thing I already know I need”.
That is the difference between buying a tool and building a brain for your business. The tool is a bet on a vendor. The system is a plan you own that whoever builds it has to work inside.
Where To Start
If you are mid-way through a decision and want a second opinion on the scope in front of you, book a 30 minute Discovery Call. No pitch, no proposal on the call. Bring the quote you have been sent and the problem you are trying to fix, and I will tell you straight whether the two match up.
Book a 30 minute Discovery Call
And if you want to sanity-check the other side of it, the Revenue Recovery Calculator will show you roughly what is sitting in your dormant database right now, which is often the fastest thing to fix regardless of who you hire.
Frequently Asked Questions
How many references should I ask an AI automation company for?
Two is enough if you pick them well. Ask for one client finished in the last three months and one from more than a year ago. The recent one tells you what the build process feels like right now. The older one tells you whether the system survived a busy quarter and whether the provider still answers the phone once the invoice is paid.
Is it rude to ask a vendor for client references?
No, and any vendor who acts like it is has told you something valuable. Reference checking is standard practice in every other significant purchase a business makes. Frame it as part of your process rather than as suspicion about them specifically, name the format and the time commitment, and most providers will organise it within a couple of days.
What questions should I ask an AI automation reference?
Five cover most of it. What did you buy and what did you end up with. What broke and how long did it take to fix. Who did the work and did you always know who to call. Are you still using it and has it changed since launch. What would you do differently if you started again. That last question produces the most useful answer almost every time.
What if the AI automation provider has no references at all?
Being new is not disqualifying, hiding it is. If there are no references, replace that assurance with structure. Stage the payments against milestones with a small, fast first deliverable. Ask them to show you the system they have built for their own business, live rather than in slides. Get ownership of accounts, data and workflows written into the agreement.
Can I trust written testimonials instead of a reference call?
Treat them as colour, not evidence. Testimonials are collected at the happiest moment of a relationship and edited before publication, so they tell you the outcome was good but nothing about how the work actually went. A live 15 minute call surfaces the response times, the handover, and the things that broke, which is the material you are actually buying.
Should the provider be on the reference call with me?
No. An email introduction is normal and helpful, but a provider sitting in on the call removes the candour that makes it worth doing. Decline politely and see how they respond. If they insist, treat that as part of your assessment rather than a scheduling detail, because it suggests the unsupervised version of the conversation would go differently.
How long should a reference call take?
Fifteen minutes. Say fifteen when you ask, keep to it, and finish on time even if the conversation is going well. Short and specific gets you a yes from busy people and produces sharper answers, because both of you stay focused. Offer to return the favour at the end, which turns the call into a fair exchange rather than a favour extracted.
About Octavius
Titus Mulquiney is the founder of Octavius AI, where he builds AI brains and AI workforces for founder-led businesses stuck running everything out of their own head. Twenty years in marketing, ex-Sony product manager, ex-GM Zeal NZ. Based in Auckland, working with operators across NZ, Australia, and the US. Connect on LinkedIn.