When vendor benchmarks are not sufficient, you need an internal evaluation. This module covers sample size, confounders, controls, and what makes a study trustworthy enough to act on.
Vendor benchmarks measure vendor-chosen tasks. Your deployment context is not a vendor-chosen task. At some point in a serious AI procurement or deployment process, you need your own data. The question is how to generate it in a way that is actually trustworthy.
An internal AI evaluation does not need to meet the standards of an academic study. It does need to meet a simpler threshold: the results should be stable enough that a different team member running the same evaluation would reach the same conclusion.
Your evaluation needs to be trustworthy enough to act on, not perfect enough to publish. The minimum viable standard: if you ran the evaluation again next week on a different sample, would the conclusion change? If yes, the evaluation is too noisy to decide anything.
The most common internal evaluation failure is a vague task definition. "Test how well the model handles customer emails" is not a task. "Classify customer support emails into five categories: billing, technical issue, feature request, complaint, and other : with an error rate below 5% on ambiguous emails from our Q3 archive" is a task.
Before any model touches your data, write down:
What exactly is the model receiving? Format, length constraints, source, any preprocessing. The more specific, the more reproducible.
What is a correct output? Who decides? If human evaluators decide, how will they score? Write the rubric before seeing any outputs.
What number, produced by this evaluation, means you proceed? Define it before running the study. Post-hoc threshold selection is the evaluation equivalent of cherry-picking.
What would a fail look like? If you cannot describe failure concretely, the success threshold is not calibrated against anything real.
Sample size determines whether your results are stable. Too small, and the conclusion could flip with five more examples. The right number depends on the accuracy level you are trying to detect and how much variance you are willing to tolerate.
A practical rule for enterprise evaluations: if you are trying to detect a difference of 10 percentage points or more, 100-200 examples is usually sufficient. If you need to detect a 5-point difference, you need 400+. If you need to detect a 2-point difference, you are conducting academic research, not an enterprise pilot.
Running 20 examples and declaring a winner. At n=20, a model that is truly 10 points better will not reliably appear better due to sampling noise. You will get the wrong answer roughly 30% of the time with that sample size.
A confounder is a variable that affects your outcome measurement but is not the thing you are trying to evaluate. In AI evaluations, the most common confounders are:
| Confounder | How it appears | Control method |
|---|---|---|
| Prompt variation | Model A gets a more specific prompt than Model B | Use identical prompts for all models |
| Sample selection | Easy examples in the test set inflate apparent accuracy | Random sample from real production cases |
| Evaluator bias | Human raters score outputs higher when they know which model produced them | Blind evaluation: raters do not know which model output they are scoring |
| Temporal shift | Test data from six months ago does not reflect current input distribution | Sample from the most recent 4-8 weeks of production inputs |
| Temperature variation | Different temperature settings change output quality inconsistently | Fix all inference parameters across all model comparisons |
How you measure correct versus incorrect determines what your evaluation actually captures. There are three measurement approaches, each with different tradeoffs.
Rule-based or model-based scoring at scale. Fast and reproducible, but can diverge from human judgment. Best for tasks with unambiguous correct answers: classification, extraction, calculation. Examples: exact match, F1 score, ROUGE.
Expert raters score outputs against a rubric. Expensive and slow, but captures nuance. Required for tasks where correctness is context-dependent: summarization quality, tone appropriateness, legal risk assessment. Must use blind evaluation and inter-rater reliability checks (Cohen's kappa should exceed 0.6 for the evaluation to be credible).
A separate model scores outputs from the model being evaluated. Faster than human evaluation, cheaper than both. Subject to positional bias (prefers the first option) and self-preference (a model scores its own outputs higher). Liang et al. (arXiv:2211.09110) document these biases in detail. If you use LLM-as-judge, use a model from a different provider than the one being evaluated, and randomize the order of options presented.
This is the most counterintuitive step. Before you run the evaluation, write out: what success looks like, what failure looks like, and what you will do in each case. This eliminates the most common form of internal evaluation bias: deciding what the results "really mean" after you see them.
If the evaluation comes back ambiguous (results are close to the threshold), that is also informative. It means the difference between models is not large enough to justify a decision based on this evaluation. You either need a larger sample or a more discriminating task.
Write down your decision rule before you see the data: "If Model A exceeds 80% on this task with a sample of 150 cases, we proceed to pilot. If not, we evaluate the next candidate." This removes the largest source of evaluation bias from internal studies.
You can now design an internal AI evaluation that is trustworthy enough to act on. The next module covers the organizational side: how to commission AI research from a team and what deliverables to require.
Precise task definition, sample size estimation, confounder control, three measurement approaches, and the pre-registration principle.
Commissioning AI Research from a Team: the 10 questions that separate a credible research brief from a vague ask.