Don't trust demos. Score vendors on the things they won't show you.
A demo is a controlled environment. Your business isn't. This scorecard tests vendors on the criteria that predict real-world success — not the ones that look good in a 45-minute presentation. Score each vendor 0–5 on every criterion. Any criterion scoring 0 is a potential knockout regardless of total score.
Vendor names
Vendor A
Vendor B
Vendor C
Criterion
Weight
What to look for
Score A
Score B
Score C
Data handling & security
Data storage & residency
Where does your data go? Who can access it? Is it used to train their models?
3×
Vague on data use
Opt-out available
No training use, documented
SOC 2 / security certifications
Do they have third-party security validation? Not just a promise — a certificate.
3×
No certifications
In process / partial
SOC 2 Type II or equivalent
Data deletion & portability
Can you get your data out? Can you delete it? What happens when you cancel?
2×
No clear process
Manual, on request
Self-serve, documented SLA
Integration & implementation reality
Integration proof (not a demo)
Have they integrated with your actual stack before? Can they show you a reference client — not a demo environment?
3×
No comparable reference
Demo only, different stack
Live reference, similar setup
Realistic implementation timeline
Ask: how long did your last three comparable implementations actually take? "Days" is almost always wrong.
3×
"Days" or "plug and play"
Weeks, vague dependencies
3–9 months, honest scoping
Data requirements disclosed upfront
Did they ask about your data quality before quoting? Or did they assume everything is clean?
2×
No data questions asked
Mentioned, not assessed
Full data assessment in scope
Cost transparency
TCO transparency
Did they proactively model data prep, training, and maintenance costs — or just show you the license fee?
3×
License only
Vague "implementation" add
Full TCO model provided
Contract exit terms
What are the cancellation terms? What's the minimum commitment? What happens to your data on exit?
2×
Annual lock-in, vague exit
Clear terms, complex exit
Month-to-month or clean exit SLA
Error handling & failure modes
Failure mode transparency
Ask them: walk me through three scenarios where this gets it wrong. What happens? Who catches it? A vendor who can't answer this hasn't thought about your use case.
3×
Can't or won't answer
Generic accuracy claims
Specific failure modes + mitigations
Messy input performance
Did you test with your actual messy data, edge cases, and exceptions — not their sample data?
2×
Demo only, clean inputs
Some edge case testing
Tested with your actual data
Support & sustainability
SMB-appropriate support tier
Are you in the enterprise tier with enterprise SLAs, or the bottom tier with community forums and email queues?
2×
Community / email only
Chat, no SLA
Named CSM or SLA <4 hrs
Vendor stability & longevity
How long have they been operating? Are they VC-dependent? What happens if they're acquired or shut down?
2×
<12 months, seed-stage
Series A, limited track record
3+ years, revenue-funded or profitable
Weighted total score
—
/ 115 max
—
/ 115 max
—
/ 115 max
Knockout criterion triggered:
A score of 0 on any criterion signals a fundamental gap — regardless of weighted total. Address this before proceeding.
Verdict
Vendor A
—
Not scored
Enter scores above and click calculate.
Vendor B
—
Not scored
Enter scores above and click calculate.
Vendor C
—
Not scored
Enter scores above and click calculate.
Notes & next steps
Before you score a vendor: run the free AI Blind Spots Check to confirm the internal readiness gaps that no vendor can solve for you. A high-scoring vendor still fails if your process foundation isn't ready.