A matrix for comparing AI tools on the criteria that actually matter — data handling, verification support, integration, cost, support and exit — rather than on features. It is the practical companion to AI Tool and Vendor Due Diligence.
The point of the matrix is to make the comparison deliberate. Most tool decisions are made on a feature list and a demo, and most of the risk sits in the columns a demo does not show.
The tool is not the risk. The retention clause is the risk.
The criteria
Score each tool from 1 (weak) to 5 (strong) on seven criteria.
| # | Criterion | What to assess |
|---|---|---|
| 1 | Data handling | Retention period, training use, processing jurisdiction, vendor-side access |
| 2 | Verification support | Whether the tool cites sources, shows its working, or supports a review step |
| 3 | Integration | Whether it connects to your system of record, and how data flows |
| 4 | Output control | Whether output format, length and structure can be constrained |
| 5 | Cost | Subscription, per-seat pricing, usage limits and the true cost at your volume |
| 6 | Support and reliability | Responsiveness, uptime, and whether there is a named person to contact |
| 7 | Exit | Whether you can export your data and prompts and delete your content |
Where a tool scores 1 on data handling, it should not be used for confidential material regardless of the total. Multiply rather than add where data handling is involved, because a fatal weakness is not offset by good features.
A worked comparison
Three tools assessed for a document-drafting workflow in a professional services firm.
| Criterion | Tool A | Tool B | Tool C |
|---|---|---|---|
| Data handling | 5 — training disabled, retention set, DPA available | 3 — business tier retains for 30 days | 2 — consumer tier, trains on input |
| Verification support | 4 — cites sources | 4 — cites sources | 3 — citations unverified |
| Integration | 3 — file upload | 5 — connects to the document system | 3 — copy and paste |
| Output control | 5 — templates and constraints | 4 — structure instructions | 3 — general |
| Cost | 4 | 3 | 5 — free |
| Support | 4 | 4 | 1 — none |
| Exit | 5 — full export | 4 — export available | 2 — no export path |
| Use | Approved for client work | Approved for internal work | Not approved |
The free tool scores highest on cost and is excluded. That is the point of the matrix: it makes the reason explicit, and it converts an argument about tools into a decision about data.
How to use it
- Assess the tools you already use first. Most businesses discover that one or two are not suitable for the work they are being used for.
- Ask for the specific clause, not the vendor’s summary, on retention and training.
- Score before the demo. A good demo moves scores that should not move.
- Record the date and the decision, so the assessment is evidenced and can be revisited.
- Re-score annually, and after any change in the vendor’s terms.
The evidence to keep
Four artefacts, for each approved tool.
- The completed matrix, with the date.
- The vendor’s own terms on retention, training and processing.
- The decision, and the categories of data the tool is approved for.
- The review date, so the assessment is revisited rather than assumed to hold.
That record answers the question a client, insurer or regulator will eventually ask: what did you assess, and when.
Frequently asked questions
What should we compare when choosing an AI tool?
Data handling, verification support, integration, output control, cost, support and exit — before features. Most risk sits in the columns a demo does not show.
Which criterion matters most?
Data handling. A tool that retains input or trains on it should not be used for confidential material, whatever else it does well.
Are free AI tools ever suitable?
For non-confidential work, sometimes. For client material, personal data or anything sensitive, use a business tier with training disabled and defined retention.
How often should we re-score tools?
Annually, and after any material change in the vendor’s terms, ownership or processing location.
Do we need a data processing agreement?
Where the tool processes personal data, yes. Many vendors offer one only on business plans, so confirm the position.
Is a formal matrix necessary for a small business?
A short version is. Even five criteria, scored honestly and recorded with a date, turns an informal decision into one you can defend.
Next step
Score the tools you already use against the seven criteria, record the date and the decision, and move any unsuitable use to an approved tool. See AI Tool and Vendor Due Diligence and Build, Buy or Partner for AI, or book an AI adoption call and we will run the assessment with you.
Sources
- Criteria reflect standard vendor-assessment practice applied to AI tools, with retention, training use and processing jurisdiction added as decisive criteria.
No statistic in this asset is invented. This asset is general information, not legal advice.