The Evaluation Theater Problem
Enterprise AI vendor evaluations follow a predictable pattern. The vendor runs a proof of concept on a curated dataset. The evaluation team scores accuracy, latency, and pricing against a scorecard. References are called. A contract is signed. Eighteen months later, the team has discovered that the accuracy numbers from the POC do not replicate on the actual production data distribution, that switching costs are substantially higher than the pricing model suggested, and that the vendor's roadmap has diverged from the organization's requirements in ways that were not visible during selection.
This is not bad luck. It is the predictable outcome of Evaluation Theater: a vendor evaluation process that measures what is easy to measure in a controlled environment, rather than what determines long-term success in an operational one.
Evaluation Theater is an AI vendor evaluation process that produces a technically rigorous-looking selection outcome while systematically failing to assess the dimensions that determine operational success and switching cost at 18-24 months post-selection. Evaluation Theater is characterized by: POC datasets that do not represent the actual production data distribution; evaluation scorecards that weight accuracy and pricing over adaptability and governance integration; and reference calls that ask about satisfaction rather than about the specific failure modes the evaluating team is most exposed to.
The Six Lock Surface Dimensions
Vendor lock in enterprise AI is not primarily a contractual phenomenon. It accumulates across six operational dimensions that standard evaluations do not assess.
Vendor Lock Surface is the complete set of operational dependencies that would require remediation if an enterprise AI vendor relationship were terminated. The Vendor Lock Surface includes data format dependencies, API contract dependencies, fine-tuning artifact portability, workflow integration depth, prompt engineering investment portability, and governance record format dependencies. An organization that measures only contractual switching costs has assessed a subset of its actual Vendor Lock Surface.
Lock Surface 1: Data Format
Training data, fine-tuning data, and evaluation data in vendor-specific formats require transformation to migrate to an alternative platform. The transformation cost is not linear with data volume; edge cases and schema differences compound.
Lock Surface 2: API Contract
Application logic that depends on vendor-specific API response schemas, streaming formats, or tool call structures requires rewriting to migrate. The depth of API contract dependency is rarely visible at evaluation time.
Lock Surface 3: Fine-Tuning Artifacts
Fine-tuned model weights in vendor-proprietary formats are non-portable. Migration requires retraining from scratch on an alternative platform, including recreating training pipelines, evaluation benchmarks, and quality gates.
Lock Surface 4: Workflow Integration
Downstream workflows that have been built around vendor-specific behavior characteristics (specific output formats, confidence score semantics, tool call patterns) require redesign if the vendor changes or is replaced.
Lock Surface 5: Prompt Investment
Prompt engineering investment is partially transferable but not fully portable. System prompts optimized for one model family's behavior patterns require rework when migrating to a different model architecture. Complex prompt engineering represents a real switching cost that is not captured in licensing fees.
Lock Surface 6: Governance Records
Governance documentation, audit trails, and compliance records in vendor-specific formats may not satisfy regulatory requirements if migrated to a different vendor's format. EU AI Act Article 12 transparency obligations require human-readable records that may be vendor-formatted.
The Three Evaluation Theater Patterns
Pattern 1: Curated Dataset POC
The vendor runs the POC on a dataset prepared by the vendor's implementation team, the evaluating team, or both, specifically to demonstrate the capability being evaluated. The dataset is cleaner, more structured, and more representative of best-case inputs than the actual production data distribution. Accuracy numbers from the POC do not replicate in production because the production distribution includes the noise, ambiguity, and edge cases that the POC dataset excluded. The correct mitigation: require the POC to run on a random sample of actual historical production queries, including the ones the team would classify as hard, before the evaluation is considered complete.
Pattern 2: Scorecard Weighting Mismatch
Standard evaluation scorecards weight accuracy, pricing, and integration capability at the top. These are the dimensions that vendors can optimize for in a short POC window. The dimensions that determine 18-month success (adaptability to edge cases, governance integration depth, roadmap alignment with the organization's specific requirements, and Vendor Lock Surface) are absent or underweighted. The correct mitigation: include Vendor Lock Surface assessment as a required scorecard section, with explicit scoring of each of the six lock surface dimensions.
Pattern 3: Satisfaction Reference Calls
Reference calls ask whether the reference customer is satisfied with the vendor. Satisfied customers do not reveal the specific failure modes the evaluating team is most exposed to, because they may not have encountered them, or because they have accommodated them and no longer experience them as problems. The correct mitigation: structured reference calls with specific questions mapped to the evaluating team's highest-risk use cases. "Have you encountered failures in [specific use case type]? How did the vendor respond?" is more useful than "are you satisfied with the vendor?"
Architecture: The Correct Evaluation Stack
Three Enterprise Scenarios
The POC Accuracy Gap
A Chief AI Officer approves an enterprise AI platform for credit analysis based on a POC that showed 91% accuracy on a curated dataset of 500 senior analyst-reviewed memos. In production, the model encounters the full distribution of credit memos, including poorly structured inputs, partial information, and cross-referenced documents. Accuracy falls to 74% within 60 days. The vendor's response: the production distribution is "different from the evaluation dataset." The evaluation team had not specified that the POC must run on a random sample of historical production inputs. The correct vendor evaluation would have required the POC to run on a stratified random sample of actual historical inputs, including the bottom 20% by structure quality. Cross-link: RAG failure modes covers how Retrieval Blindspot compounds this accuracy gap in knowledge-intensive use cases.
The Governance Record Lock
A COO deploys an AI clinical documentation assistant from a vendor whose governance record format is proprietary. Eighteen months later, a regulatory inquiry requires producing audit trail documentation for a specific class of outputs in a format compatible with the organization's existing compliance record system. The vendor's proprietary format requires a translation layer that does not exist. The organization must manually reconstruct the governance record for the inquiry period. The correct vendor evaluation would have included a Lock Surface assessment of governance record portability: can the audit trail be exported in a standard format (JSON-LD, CSV with documented schema) that integrates with existing compliance infrastructure? Red flag during evaluation: the vendor's governance documentation is only accessible through the vendor's own compliance portal.
The Prompt Engineering Switching Cost
A general counsel's team spends six months building and refining a complex system prompt architecture for contract review, including specialized instruction sets for twelve document types. The organization then decides to evaluate an alternative model provider with a substantially lower inference cost. The prompt evaluation reveals that the existing prompt architecture requires significant rework for the new model family's different behavior patterns: instruction hierarchy interpretation differs, output format specifications behave differently, and the tool call patterns in the agentic workflow require redesign. The switching cost estimate is four months of senior prompt engineering time, effectively eliminating the cost advantage of the lower-inference-cost provider. The correct vendor evaluation would have assessed Lock Surface 5 upfront: how model-family-specific is the prompt engineering investment? For complex prompt architectures, a portability assessment should be part of the initial selection criteria. Cross-link: prompt debt covers how accumulated prompt complexity compounds switching costs.
Executive Checklist: Vendor Lock Surface Assessment
Is your POC running on a production-representative data sample?
Good: POC uses a stratified random sample of actual historical production inputs, including hard cases and edge cases. Red flag: POC uses a curated dataset prepared by the vendor or the implementation team specifically for the evaluation.
Have you scored all six Vendor Lock Surface dimensions?
Good: documented lock score for each dimension, with explicit switching cost estimate. Red flag: evaluation scorecard covers accuracy, pricing, and integration without a Vendor Lock Surface section.
Are fine-tuned model weights exportable in a non-proprietary format?
Good: vendor confirms weight export in open format (GGUF, HuggingFace safetensors); contractually specified. Red flag: fine-tuning is performed on vendor infrastructure with no export mechanism.
Can governance records be exported in a standard, auditable format?
Good: audit trail export in documented schema (JSON-LD or equivalent) confirmed in vendor contract. Red flag: governance records accessible only through vendor's compliance portal, with no export API.
Are reference calls structured around your highest-risk use cases?
Good: reference call script includes specific questions about failure modes relevant to your use case type. Red flag: reference calls ask general satisfaction questions without use-case-specific failure mode inquiry.
Is the vendor's roadmap aligned with your 24-month requirements?
Good: written roadmap alignment document with specific capability commitments and delivery milestones, included in the contract. Red flag: roadmap alignment discussed verbally in sales meetings, not documented in the agreement.
Have you computed the full 24-month switching cost including all six lock dimensions?
Good: documented switching cost estimate for each lock dimension, summed to a total switching cost that is part of the selection decision. Red flag: switching cost considered only at the contractual termination fee level.
Does the vendor's governance integration meet your EU AI Act or NIST AI RMF obligations?
Good: documented mapping from vendor's governance capabilities to your specific compliance obligations, verified before contract signature. Red flag: vendor claims compliance; specific obligations and mechanisms not verified against your regulatory context.
Build, Buy, or Configure
Build
- Vendor Lock Surface scoring rubric calibrated to your use case types
- Production-representative POC dataset construction process
- Structured reference call script mapped to your failure modes
- 24-month switching cost estimation model
Buy (Vendor Category)
- AI vendor evaluation platforms with lock surface analysis
- Third-party AI vendor due diligence services
- Contract intelligence tools for AI-specific term analysis
- Multi-vendor AI governance platforms that reduce lock surface by design
Configure
- Existing vendor evaluation templates to add lock surface section
- Existing RFP process to require production-distribution POC
- Existing reference call process to add failure-mode-specific questions
Three-Phase Roadmap
Evaluation Reform
- Build Vendor Lock Surface scoring rubric for your vendor categories
- Define production-representative dataset construction criteria
- Create structured reference call script for your use case types
- Gate: updated vendor evaluation framework approved by procurement and legal
Current Vendor Audit
- Apply Vendor Lock Surface assessment to all active vendor relationships
- Compute 24-month switching cost for each vendor
- Identify highest-lock-surface relationships for strategic review
- Gate: lock surface map for all active enterprise AI vendors
Annual Vendor Review
- Annual re-assessment of lock surface for all active vendors
- Track lock surface growth as integrations deepen over time
- Include lock surface trajectory in vendor relationship health metrics
- Gate: lock surface metric in annual vendor governance review
Cost of Inaction
POC-to-Production Accuracy Gap
The cost of discovering the POC accuracy gap in production rather than during evaluation is remediation: either additional vendor customization (at vendor pricing), or a delayed re-evaluation process with the same or alternative vendors. Both costs are substantially higher than the cost of requiring a production-representative POC.
Unquantified Switching Costs
An enterprise AI vendor relationship with an unmapped Vendor Lock Surface is a liability with an unknown magnitude. Organizations that discover the lock surface depth only when they attempt to renegotiate or migrate face switching costs they could not have anticipated in the contract decision.
Governance Record Exposure
Governance records in proprietary vendor formats create a regulatory exposure that is not visible at contract time. EU AI Act Article 12 transparency obligations and NIST AI RMF governance requirements may require format conversion that the vendor does not support, at a cost that escalates with the volume of historical records requiring conversion.
Negotiating Leverage Loss
A vendor that knows its Vendor Lock Surface is deep has less incentive to negotiate on pricing, roadmap commitments, or SLA terms at renewal. Organizations that map the lock surface before contract signature can negotiate portability provisions that preserve their negotiating leverage at renewal. Cross-link: enterprise AI governance stack covers how vendor integration depth adds Control Debt to the governance surface.
Excited about AI, innovation, and growth?
Start a conversationReferences
- European Parliament and Council. "Regulation (EU) 2024/1689 on Artificial Intelligence (AI Act)." Official Journal of the European Union, 2024. Article 12 (Transparency and provision of information to deployers) and Article 26 (Obligations of deployers of high-risk AI systems).
- National Institute of Standards and Technology. "Artificial Intelligence Risk Management Framework (AI RMF 1.0)." NIST AI 100-1, 2023. Govern and Measure functions.
- Liang, P., et al. "Holistic Evaluation of Language Models." TMLR, 2023. arXiv:2211.09110. Basis for evaluation framework design.
- Bommasani, R., et al. "On the Opportunities and Risks of Foundation Models." arXiv:2108.07258, 2022. Section on ecosystem lock-in and concentration risk.
- International Organization for Standardization. "ISO/IEC 42001:2023: Artificial Intelligence Management System." 2023. Clause 8.4 on AI system lifecycle management.