name	ai-vendor-evaluation
description	Comprehensive framework for evaluating AI vendors and solutions to avoid costly mistakes. Use this skill when assessing AI vendor proposals, conducting due diligence, evaluating contracts, comparing vendors, or making build-vs-buy decisions. Helps identify red flags, assess pricing models, evaluate technical capabilities, and conduct structured vendor comparisons.

AI Vendor Evaluation

Version 1.0 | October 2025 | Based on $1.2M average AI spend analysis

Overview

This skill provides a systematic framework for evaluating AI vendors and solutions to avoid the costly mistakes that plague 95% of AI projects. Use when conducting vendor due diligence, evaluating proposals, negotiating contracts, or making strategic AI purchasing decisions.

Key capabilities:

Structured evaluation criteria for AI vendors
Red flag identification in proposals and demos
Pricing model analysis and fair market rates
Technical capability assessment
Contract term evaluation
Build vs buy decision framework

Quick Decision Tree

Start here to determine which references to read:

What stage are you in?

├─ Early exploration (multiple vendors being considered)
│  └─ Read: evaluation-criteria.md, use-case-fit.md
│     Use: scorecard-template.xlsx
│
├─ Evaluating specific proposal or demo
│  └─ Read: red-flags.md, technical-assessment.md
│     Check: pricing-models.md for pricing reasonableness
│
├─ Contract negotiation
│  └─ Read: contract-checklist.md, pricing-models.md
│     Reference: red-flags.md for problematic terms
│
├─ Build vs Buy decision
│  └─ Read: build-vs-buy.md, use-case-fit.md
│     Consider: Total cost of ownership from pricing-models.md
│
└─ Post-purchase review or audit
   └─ Read: evaluation-criteria.md, technical-assessment.md
      Assess: Whether vendor is delivering on promises

When to Use This Skill

Trigger scenarios:

"Help me evaluate this AI vendor proposal"
"What should I look for in AI vendor demos?"
"Is this pricing reasonable for an AI solution?"
"Should we build or buy this AI capability?"
"What questions should I ask this AI vendor?"
"Help me compare these AI vendors"
"Review this AI contract for red flags"
"Conduct due diligence on this AI company"

Core Evaluation Framework

Phase 1: Initial Screening

Goal: Eliminate obviously problematic vendors before deep evaluation

Key questions:

Does the vendor have relevant domain experience?
Are there verifiable customer references?
Is the technology approach sound?
Are pricing and terms transparent?

Read: references/red-flags.md for disqualifying signals
Read: references/use-case-fit.md for domain fit assessment

Phase 2: Deep Evaluation

Goal: Assess vendor capabilities systematically across all dimensions

Evaluation dimensions:

Technical capability - Can they actually deliver?
Business viability - Will they still exist in 2 years?
Pricing fairness - Are costs reasonable for value delivered?
Implementation risk - How likely is successful deployment?
Contract terms - Are legal terms acceptable?

Read: references/evaluation-criteria.md for comprehensive framework
Read: references/technical-assessment.md for technical evaluation
Read: references/pricing-models.md for pricing analysis
Use: assets/scorecard-template.xlsx to score vendors systematically

Phase 3: Contract Negotiation

Goal: Secure favorable terms and avoid costly traps

Critical areas:

Performance guarantees and SLAs
Data ownership and usage rights
Pricing structure and escalation terms
Exit clauses and data portability
Liability and indemnification

Read: references/contract-checklist.md for essential terms
Reference: references/red-flags.md for problematic contract patterns

Common Vendor Patterns

The Overpromiser

Characteristics: Claims to solve everything, vague on technical details, aggressive sales tactics
Red flag: "Our AI can handle any use case"
Response: Demand specific technical explanations and verifiable references

The Feature Dumper

Characteristics: Long feature lists, complex pricing, unclear core value proposition
Red flag: Can't explain what problem they actually solve
Response: Force clarity on primary use case and success metrics

The Consultant in Disguise

Characteristics: Software license + mandatory professional services
Red flag: Professional services cost more than software
Response: Assess true cost of ownership, consider if you're buying software or consulting

The Model Wrapper

Characteristics: Thin layer over OpenAI/Anthropic APIs with high markup
Red flag: No proprietary technology, just API access + UI
Response: Calculate cost of building similar solution in-house

Full pattern library: See references/red-flags.md

Build vs Buy Decision Framework

When to read this section: Before committing to vendor evaluation, determine if building in-house is better option.

Key factors:

Capability availability - Does suitable vendor solution exist?
Time to value - Buy: weeks-months, Build: months-years
Total cost - Consider 3-year TCO for both options
Strategic importance - Core competency? Build. Commodity? Buy.
Team capability - Do you have talent to build and maintain?

Read: references/build-vs-buy.md for detailed decision framework

Using the Scorecard Template

The vendor scorecard enables structured comparison across vendors.

To use:

Open assets/scorecard-template.xlsx
List vendors to compare (up to 5)
Score each vendor on evaluation criteria (1-5 scale)
Review weighted scores and vendor comparison chart
Document decision rationale

Customization: Adjust weights based on priorities for your specific use case.

Reference Documents

references/evaluation-criteria.md

Comprehensive scoring framework across all vendor evaluation dimensions. Includes specific questions to ask, what constitutes good/bad answers, and how to weight criteria for different use cases.

Use when: Conducting systematic vendor evaluation

references/red-flags.md

Catalog of warning signs indicating problematic vendors. Organized by category: technical red flags, business red flags, pricing red flags, contract red flags, and behavioral red flags.

Use when: Initial vendor screening or reviewing proposals

references/pricing-models.md

Guide to AI vendor pricing models (per-seat, usage-based, platform fees, etc.), fair market rates, what drives costs, and how to negotiate. Includes pricing red flags and total cost of ownership analysis.

Use when: Evaluating vendor pricing or negotiating contracts

references/technical-assessment.md

Framework for assessing technical capabilities: architecture review, model evaluation, integration complexity, scalability, security, and data handling. Includes specific technical questions to ask.

Use when: Deep technical evaluation of vendor capabilities

references/contract-checklist.md

Essential contract terms for AI vendor agreements: performance guarantees, data rights, pricing protection, exit terms, liability, and support commitments. Includes negotiation guidance.

Use when: Contract review or negotiation

references/use-case-fit.md

Framework for assessing whether vendor solution actually fits your use case. Includes questions to ask yourself, questions to ask vendor, and warning signs of poor fit.

Use when: Initial vendor screening or use case definition

references/build-vs-buy.md

Decision framework for whether to build AI capability in-house vs purchasing vendor solution. Includes total cost analysis, capability assessment, and strategic considerations.

Use when: Before committing to vendor evaluation process

Assets

assets/scorecard-template.xlsx

Structured spreadsheet for vendor comparison with:

Evaluation criteria organized by category
Scoring system (1-5 scale) with descriptions
Weighted scoring based on priorities
Vendor comparison charts
Decision documentation section

Customize: Adjust criteria weights and add company-specific requirements

ai-vendor-evaluation

Install Skill

SKILL.md