vendor-evaluation ยท diff
git:20260718.7c1285c to git:20260728.fac4544
31 added, 28 removed. Audit A to A.
---
name: vendor-evaluation
description: Evaluate vendors against weighted criteria, a proof of concept that tests your real workload, and a clear-eyed accounting of exit costs before you sign. Use when choosing a paid tool or service you will depend on and a wrong pick is expensive to undo.
---
# Vendor evaluation
Choosing a vendor is choosing a dependency you will live with for years and pay
to leave. The evaluation goes wrong when it is run on demos and vibes: the
polished sales walkthrough decides it, criteria get invented to justify the
- favorite, and no one asks what it costs to get back out. A disciplined evaluation
- tests the vendor on your work and prices the exit before it prices the entry.
+ favorite, and no one asks what it costs to get back out. A disciplined
+ evaluation tests the vendor on your work and prices the exit before it prices
+ the entry.
## Method
1. **Set weighted criteria before you see a single demo.** List what matters:
- functionality, reliability, security posture, support, total cost, and how hard
- it is to leave. Assign weights that sum to a fixed total, and write them down
- first. Criteria invented after the demo just rationalize the vendor who demoed
- best.
+ functionality, reliability, security posture, support, total cost, and how
+ hard it is to leave. Assign weights that sum to a fixed total, and write them
+ down first. Criteria invented after the demo just rationalize the vendor who
+ demoed best.
2. **Score against requirements, not against each other.** Rate every vendor on
- each criterion against your bar, so a weak field cannot make a mediocre option
- look strong by comparison. Separate must-haves, which are pass or fail, from
- nice-to-haves that earn weighted points.
- 3. **Design the proof of concept around your real workload.** Feed it your actual
- data shapes, your peak volume, your awkward edge cases, and your integration
- points. A POC on the vendor's sample data proves their demo works. Define the
- success metrics and the time box before you start so the POC ends in a verdict,
- not an open-ended trial.
+ each criterion against your bar, so a weak field cannot make a mediocre
+ option look strong by comparison. Separate must-haves, which are pass or
+ fail, from nice-to-haves that earn weighted points.
+ 3. **Design the proof of concept around your real workload.** Feed it your
+ actual data shapes, your peak volume, your awkward edge cases, and your
+ integration points. A POC on the vendor's sample data proves their demo
+ works. Define the success metrics and the time box before you start so the
+ POC ends in a verdict, not an open-ended trial.
4. **Cost the exit, not just the entry.** Estimate what it takes to migrate off:
- data export formats, proprietary interfaces you would rewrite, retraining, and
- contract lock-in. High switching cost is a real price even if it never appears
- on an invoice. Prefer vendors whose data you can get back in a usable form.
- 5. **Verify security and compliance with evidence, not the questionnaire alone.**
- Ask for the SOC 2 or ISO report, the data processing terms, subprocessor list,
- breach history, and where data is stored. A vendor who cannot produce these for
- a serious deal is telling you something.
+ data export formats, proprietary interfaces you would rewrite, retraining,
+ and contract lock-in. High switching cost is a real price even if it never
+ appears on an invoice. Prefer vendors whose data you can get back in a usable
+ form.
+ 5. **Verify security and compliance with evidence, not the questionnaire
+ alone.** Ask for the SOC 2 or ISO report, the data processing terms,
+ subprocessor list, breach history, and where data is stored. A vendor who
+ cannot produce these for a serious deal is telling you something.
6. **Reference-check past the list they gave you.** Talk to the references, then
- find a customer they did not hand you and ask what breaks at month six: support
- response times, hidden costs, the feature that was "on the roadmap." Sales sells
- the roadmap; a real customer tells you the product.
+ find a customer they did not hand you and ask what breaks at month six:
+ support response times, hidden costs, the feature that was "on the roadmap."
+ Sales sells the roadmap; a real customer tells you the product.
## Checks
- Were the weighted criteria fixed before demos, or shaped to fit a favorite?
- Did the POC run on your workload and data, or on the vendor's canned scenario?
- Can you state, in dollars and weeks, what leaving this vendor would cost?
## Boundaries
- This picks the vendor; it does not negotiate the contract terms or run procurement
- and legal review, which have their own owners and thresholds. Approval limits and
- required security sign-offs are company policy: route spend and data agreements
- through the process your organization requires rather than deciding them here.
+ This picks the vendor; it does not negotiate the contract terms or run
+ procurement and legal review, which have their own owners and thresholds.
+ Approval limits and required security sign-offs are company policy: route spend
+ and data agreements through the process your organization requires rather than
+ deciding them here.