What the data showed
Same trials. Same varieties. Opposite winner, depending entirely on who published the result.
Growth Academy · Section 5 · Running the machine
If the vendor sitting across from you were paid only when you got the right answer, would this pitch look the same?
Most vendor calls follow the vendor's script: a features tour, a report view, a happy path built to look good. If you sit through it passively, you learn whether the product is polished. You learn nothing about whether it handles the parts of your business that are actually difficult.
One operator, evaluating a booking-and-ticketing platform for a client running an inflatable water park in Dubai, refused to sit through the vendor's script. She interrupted it, repeatedly, and forced it onto the client's actual operations, what a guest sees on the front endThe part of software users see and click. versus what runs on the back endThe behind-the-scenes machinery of software that stores the data and does the work., and who signs what when one person books for a group: is this the front end or the back end, what does the customer see versus the admin; the buyer signs the waiver, but the other four guests on the booking aren't on this call, how do they sign; you're showing me the report, I'm not interested in the report, I want to see how we connect. Each interruption turned a feature tour into a test of whether the tool survived contact with a real edge case.
The same discipline applies to any vendor, not only software. On a first call with a new marketing vendor, one buyer stopped a generic platform walkthrough and asked for something concrete instead: examples of finished work, what a real deliverable looks like end to end, where the platform's capability actually stops.
"We bought into the possibility of what can this platform give to us. Right now, we have no idea what level of delivery, what sort of project... Is there something like this you can walk us through? Here's what you can expect, here's examples of the work that we've done."
Both moves share a mechanism: a generic walkthrough tells you the platform is usable, it does not tell you what you can safely commit real deadlines to. The fix is the same either way. Write your real workflow down as ordered steps before the call. Walk it through the product yourself. Where the vendor shows you an output, ask to see the setup path that produced it. Output views hide labor.
Feature lists answer what a platform does. They never answer who carries the operational burden, which is where hidden cost, delay, and dependency actually live. Watching the same demo, the same operator cut through a marketplace-integration slide to ask the one question that mattered.
Before committing to any platform, force an explicit division of labor for every integration and workflow it promises to handle. The vendor's co-founder was walking the same operator through marketplaceA platform where many independent sellers offer products to the platform's buyers. integrations, travel platforms, channel partners, each one presented as a solved feature. She cut past the feature list to the one question that exposes what a feature list hides.
"Are we doing those connections, or are you guys — you will handle those on our behalf?"
The answer changed the evaluation. Each marketplace, it turned out, required its own one-by-one integration rather than a single channel managerOne tool that pushes your listings or inventory out to many sales platforms at once., with the client providing an account and API keyA private code that proves your software is allowed to use another company's service. while the vendor's back end managed the connection. That is estimable. "Yes, it integrates" is not. Ask this question the moment any capability is demonstrated, especially when what you're shown is the output, a report, a connected channel, rather than the setup path.
A technical co-builder in agriculture data pulled hundreds of thousands of field trials comparing two competing seed brands, run in the same fields, in the same years. He split the results by who had reported them.
Same trials. Same varieties. Opposite winner, depending entirely on who published the result.
"Whenever [the first company] was reporting the data, they beat [the competitor] 90% of the time... whenever [the competitor] reported on their test with [the first company], same variety, they won 90% of the time. Please explain to me how that is possible. Same test run by two different people, and you get different answers 90% of the time."
The mechanism underneath is structural, not a case of anyone lying. A seed company is designed to sell seed, not to help a farmer find the objectively best seed for their field. Their trial data, their recommendations, their sales incentives all point toward moving their own product line. The discriminator: does the party handing you the evidence profit specifically from one conclusion over another? If yes, treat what they show you as marketing-shaped evidence, not neutral evidence, and expect it as a known category of bias to correct for, since these internal tests are typically not even designed for fair comparison in the first place. It generalizes past agriculture, to software benchmarks, sponsored case studies, and a "free consultation" that ends in a proposal from the consultant's own firm.
Two more moves round out the interrogation. When inheriting a decision made by a prior vendor or agency, ask whether it was the vendor's choice or the client's direction, since the two carry very different signal about whether the choice still fits your situation. And before signing anything, ask the vendor directly what went wrong for past clients, not just for their wins, because the pattern in that answer, a stage mismatch, targeting that ran too broad, an account-team fit problem, tells you more about your real risk than any testimonial.
Everything above is diagnosis. This is what you do with what you learn: vet the engagement by the outcome it promises, define success in writing before you sign, and protect your own cash and reputation from the fuzzy parts of the deal.
One more discriminating question sits underneath all of this: should you even be buying, or should you build it yourself? One rule, from a water-park ticketing evaluation, holds up well: buy a dependable tool with real support behind it, and build only when the technology is the actual business. "We don't want to create our own tool. I'm not in the process to build a tech company here. We have the business." Two non-negotiables when you buy instead of build: something you can implement fast, and a vendor you can depend on when it breaks.
Everything on the last three pages is a lens. This turns the lens on a vendor you are actually evaluating right now, or one you recently bought from.
Pick one tool or service you are currently evaluating, or recently bought.
You did it right if your qualifying event is measurable by you, without the vendor's own dashboard, and it exists in writing before money moves.
Vendor Evaluation Scorecard. The drill above, running as an instrument: it will refuse to score a vendor until you've entered at least three real workflow steps, it caps or zeroes any evidence you can't attribute to an independent source, and it won't show you an overall score until your qualifying event passes a measurability check. Ships alongside this section.
Next in this section: the pilot reframe. Funding a paid engagement honestly, separating what you're proving from what you're absorbing on spec, and the two questions that belong in your next capital conversation.