Module 2: Choosing AI Vendors · AI for Enterprise Leaders

In 2022, a CTO signed a three-year, $4 million AI platform deal. Two years in, half the promised features had not shipped, the roadmap had changed three times, and the contract had a minimum spend commitment that made it prohibitively expensive to switch. That deal was not unusual. This module gives you the framework to avoid repeating it.

Build vs Buy vs Partner

Every AI capability decision starts with the same question: should you build it internally, buy a vendor solution, or partner with a systems integrator who will assemble the pieces? The wrong choice locks you into a path that is expensive to reverse.

Build internally when the capability is a genuine competitive differentiator, when your data is unique and provides an irreplaceable advantage, or when security requirements prohibit sending data to external systems. Building internally gives you control but requires sustained engineering investment and takes longer to deliver initial value.

Buy a vendor solution when a mature market exists with multiple competing vendors solving your problem, when the capability is not differentiating (expense reporting, meeting summarization, document parsing), or when you need to show value quickly and cannot afford a multi-month build cycle. Buying is faster but creates dependency on the vendor's roadmap and pricing decisions.

Partner with a systems integrator when you need custom integration with complex existing systems, when you lack internal AI engineering capacity, or when the use case sits at the intersection of domain expertise and AI capability that no single vendor addresses. Partnerships give you flexibility but require careful governance to avoid capability leakage to the partner over time.

The hybrid trap Many organizations try to do all three simultaneously: build some things, buy others, and partner for integration. This is sometimes necessary, but it multiplies governance complexity. Every vendor relationship, internal build, and partner engagement is a dependency. Map them before you commit, not after.

The 7-Dimension Vendor Scorecard

A vendor who wins a demo does not necessarily win a production evaluation. Demos are designed to show the best case. Production deployments reveal the real case. Evaluate every AI vendor across these seven dimensions before signing a contract.

Capability

Does the product solve your specific problem at your scale? Test with your data, not the vendor's demo data.

Reliability

What is the uptime SLA? What is the escalation process for outages? Ask for the last six months of incident reports.

Security

SOC 2 Type II certification, data retention policies, and whether your data is used for model training. This is non-negotiable for enterprise data.

Integration

Does it connect to your existing systems? What is the API maturity? Is there an enterprise-grade SDK or only a REST endpoint?

Pricing Model

Is pricing predictable at scale? Token-based or API-call pricing can produce billing surprises at 10x expected volume. Model this before signing.

Lock-in Risk

Can you export your data and configurations? Is your fine-tuned model portable? What happens if the vendor raises prices in year two?

Support Quality

Is there a named customer success manager for enterprise accounts? What is the response time SLA for critical issues? Test this during the pilot.

Red Flags in AI Vendor Demos

Every AI vendor demo is a best-case performance. The task was designed to succeed. The data was curated in advance. The edge cases were removed. Here are the five red flags that signal the production reality will look very different from the demo.

Red flag 1: Cherry-picked benchmarks. "Our model achieves 94% accuracy on benchmark X" tells you nothing about accuracy on your data in your domain. Ask the vendor to run the evaluation on a sample of your own data before the demo ends.

Red flag 2: Vague roadmap commitments. "That feature is on our roadmap for Q3" is not a commitment. Ask for the feature in writing as a contract deliverable with a penalty clause for non-delivery. If they refuse, assume it will not ship.

Red flag 3: No reference customers in your industry. AI systems that work in consumer social media may perform very differently in regulated financial services or clinical settings. Ask for three reference customers in your specific industry and call them before signing.

Red flag 4: Inability to explain failures. Ask the vendor to show you examples where the system got the wrong answer. How the vendor responds to this request is more informative than the demo itself. A mature vendor will have a documented failure mode catalog. A less mature one will struggle to answer.

Red flag 5: Resistance to a pilot before a multi-year contract. A vendor who asks you to sign a three-year minimum spend commitment before a paid pilot is protecting their sales cycle, not your interests. Insist on a 90-day pilot with well-defined success criteria before any multi-year commitment.

The Three Contract Traps

Even after a successful vendor evaluation, the contract can create dependencies that are difficult and expensive to exit. Know these three traps before your legal team reviews the draft.

Trap 1: Minimum spend commitments. A contract that commits you to paying a minimum monthly or annual fee regardless of usage is a minimum spend commitment. This is common in enterprise AI contracts and creates a hostage situation: if the system underperforms, you still owe the money. Negotiate minimums down or out, and if you accept them, make them conditional on the vendor meeting performance SLAs.

Trap 2: Data usage clauses that allow model training on your data. Some AI vendors include clauses permitting them to use your data to train or improve their models. This is a significant IP and confidentiality risk. Read these clauses carefully. The acceptable version: your data is processed to serve your requests and deleted according to a stated retention policy. The unacceptable version: your data may be retained and used to improve products that are then sold to your competitors.

Trap 3: Model deprecation without migration support. AI models are updated and deprecated on cycles that are outside your control. A vendor can retire the model version your workflows depend on with 30-90 days of notice, forcing an emergency migration. Negotiate for a minimum deprecation notice period (12 months is reasonable), migration support services at no additional cost, and the right to continue using a deprecated model for up to 12 months after EOL.

Python · Vendor Scorecard

import csv
from dataclasses import dataclass, field
from typing import Dict

@dataclass
class VendorDimension:
    name: str
    weight: float  # weights should sum to 1.0
    score: float = 0.0  # 0-10

SCORECARD_DIMENSIONS = [
    VendorDimension("Capability", weight=0.25),
    VendorDimension("Reliability", weight=0.20),
    VendorDimension("Security",    weight=0.20),
    VendorDimension("Integration", weight=0.15),
    VendorDimension("Pricing",     weight=0.08),
    VendorDimension("Lock-in Risk",weight=0.07),  # lower score = higher risk
    VendorDimension("Support",     weight=0.05),
]

def weighted_vendor_score(dims: list[VendorDimension]) -> float:
    """Return 0-10 weighted vendor score."""
    return sum(d.score * d.weight for d in dims)

def compare_vendors(vendors: Dict[str, list[float]]) -> None:
    """Print ranked comparison of vendors."""
    results = []
    for vendor_name, scores in vendors.items():
        dims = [VendorDimension(d.name, d.weight, s)
                for d, s in zip(SCORECARD_DIMENSIONS, scores)]
        total = weighted_vendor_score(dims)
        results.append((vendor_name, total))
    results.sort(key=lambda x: x[1], reverse=True)
    for rank, (name, score) in enumerate(results, 1):
        print(f"{rank}. {name}: {score:.1f}/10")

# Example:
compare_vendors({
    "Vendor A": [8, 9, 7, 6, 7, 8, 9],
    "Vendor B": [9, 7, 9, 8, 5, 6, 7],
    "Vendor C": [7, 8, 8, 9, 8, 9, 8],
})

Try this

Take one AI vendor your organization is currently evaluating or has evaluated in the past. Score them on each of the seven dimensions using a 0-10 scale. Then call one of their reference customers and ask specifically about reliability, support quality, and any surprises in the contract. What dimension had the biggest gap between the demo impression and the reference customer's experience?