Human Evaluation in AI: Designing Rubrics and Managing Annotators

Why Human Evaluation Cannot Be Skipped

Every automated metric for AI output quality is a proxy. BLEU score proxies translation quality through n-gram overlap. Accuracy proxies task success through agreement with reference labels. Perplexity proxies language model quality through probability of held-out text. These proxies are useful and scalable, but they diverge from human judgment in systematic ways that matter for deployment.

The divergence is most pronounced for open-ended tasks. A model that generates a factually correct but unhelpfully organized answer to a user's question may score well on accuracy while failing the user. A model that produces a grammatically correct but culturally inappropriate response may score well on fluency metrics while creating real harm. These failures are invisible to automated metrics and visible to humans.

Zheng et al. (arXiv:2306.05685) studied the correlation between human preferences and automated metrics across a range of text generation tasks and found significant divergence in multiple task categories. The study demonstrated that neither approach alone is sufficient: automated metrics provide coverage and speed, human evaluation provides validity and nuance, and the combination is more reliable than either alone.

Rubric Design: The Foundation of Consistent Evaluation

The quality of human evaluation depends on the quality of the rubric. A rubric is a structured guide that tells evaluators exactly what to look for, how to score it, and what examples look like at each score level. Without a well-designed rubric, individual annotators apply their own criteria, and the resulting scores are not comparable across annotators or over time.

Effective rubrics have four properties. First, they are specific to the task: a rubric designed for medical question answering looks different from a rubric for customer service chatbot evaluation, because the relevant quality dimensions differ. Second, they define 3 to 5 dimensions, because this is the range where annotator attention is manageable. More dimensions than five spread attention too thin and reduce the reliability of each individual score. Third, each dimension has a defined scale with anchored examples: what does a score of 1, 3, and 5 look like on the Accuracy dimension? Anchored examples reduce subjectivity more than abstract definitions. Fourth, rubrics address the specific failure modes that matter for the deployment context, including the safety-critical ones that would be disqualifying even if everything else looked good.

Here is an example rubric structure for factual accuracy in a question-answering context:

Selecting and Training Annotators

Annotator selection has a larger effect on evaluation quality than most teams realize. The profile of annotators should match the profile of the users the model will serve, not the profile of the team building the model. A technical team evaluating a customer service model designed for general consumers will systematically underweight failures that affect non-technical users, because technical evaluators approach the model differently and have different tolerance for indirect or imprecise responses.

For domain-specific applications, domain expertise matters. A medical information model requires annotators with clinical knowledge who can identify factual errors, appropriate hedging, and contraindicated advice. A legal drafting model requires annotators who understand legal language and can identify when clauses are ambiguous or absent. General-purpose annotators cannot substitute for domain expertise in these contexts, regardless of how clear the rubric is.

Annotator training should include three components. First, an introduction to the rubric with worked examples at each score level. Second, a practice set where annotators score examples and receive feedback on how their scores compare to the ground truth or to expert consensus. Third, a calibration session where multiple annotators score the same set of examples and discuss disagreements until they reach convergence on how to apply the rubric. This calibration is not about forcing agreement. It is about ensuring annotators understand the rubric's intent well enough to apply it consistently.

The reliability of human evaluation depends more on rubric clarity and annotator calibration than on the number of annotators or the scale of the evaluation.

Inter-Annotator Agreement: Measuring Evaluation Reliability

Inter-annotator agreement (IAA) measures how consistently different annotators score the same item. High IAA means the rubric is being applied consistently across annotators. Low IAA means annotators are interpreting the rubric differently, and evaluation scores are not comparable across annotators.

Cohen's kappa (Cohen, 1960) is the most widely used IAA metric for categorical ratings. It adjusts for chance agreement: two annotators choosing randomly from a 4-point scale would agree 25% of the time by chance. Cohen's kappa measures agreement above chance. Scores in the substantial agreement range (roughly 0.61-0.80) are generally considered the minimum for evaluation data to be reliable enough for model comparison or quality assessment. Below the moderate agreement threshold, the evaluation is unreliable: scores from different annotators are not measuring the same thing.

For ordinal scales, weighted kappa penalizes disagreements proportionally to how far apart the scores are. Krippendorff's alpha generalizes to multiple annotators and various measurement scales. The choice of metric matters less than the practice of measuring IAA at all and acting on it when it falls below acceptable thresholds.

When IAA is low, the response is not to average the scores and move on. It is to investigate the source of disagreement. Are annotators interpreting a specific dimension differently? Does a category of examples have genuinely ambiguous quality that the rubric does not address? Are some annotators applying the rubric more strictly than others? The answers to these questions lead to rubric improvements or annotator retraining that produce more reliable evaluation going forward.

Calibration Sessions and Their Role

Calibration sessions bring annotators together to score the same set of examples and discuss disagreements. They are the most effective mechanism for improving IAA and ensuring the rubric is applied consistently across an annotation team.

A calibration session typically works as follows. Each annotator scores the same 20 to 40 examples independently, without seeing others' scores. Scores are then revealed and disagreements identified. For each significant disagreement, annotators discuss their reasoning: why did they score this example differently? What does the rubric say, and does the rubric need to be clarified to address this type of example? The session ends with updated rubric notes and a shared understanding of edge cases that were not well-specified originally.

Calibration sessions should happen at the start of a new evaluation project, when new annotators join an existing project, and periodically during long-running evaluation programs to prevent annotators from drifting in their interpretations over time. The investment per session is modest relative to the value of reliable evaluation data.

Scaling Human Evaluation

Human evaluation is inherently limited in scale compared to automated metrics. A team of five annotators working full time can score perhaps 500 items per day. This is sufficient for structured evaluation projects and for sampling from deployment traffic, but it cannot match the volume of automated evaluation. The solution is not to replace human evaluation with automated metrics. It is to use both strategically.

Human evaluation should be used for tasks where automated metrics are known to diverge from human judgment, for validating that automated metrics are behaving as expected, for adversarial and edge-case examples where automated metrics are least reliable, and for periodic quality checks on samples from larger automated evaluation runs. Automated metrics should be used for high-volume routine quality monitoring where human evaluation at the same scale would be impractical.

LLM-based evaluation, where a language model judges the quality of another language model's output, is an increasingly common way to scale evaluation beyond what human teams can cover. It is faster and cheaper than human evaluation, and research has shown that LLM judges can achieve reasonable correlation with human preferences on some tasks. However, Zheng et al. demonstrated that LLM judges introduce their own biases, prefer verbose responses, and can be manipulated by surface features of the output. LLM-based evaluation supplements human evaluation rather than replacing it, particularly for tasks with well-defined quality criteria where LLM judgment is better calibrated.

The Full Evaluation Stack

A complete evaluation program for a deployed AI model combines human evaluation with automated metrics in a layered stack. Each layer serves a different purpose and operates at a different scale and frequency.

The base layer is automated metrics running on all or nearly all outputs, checking for basic quality signals such as safety classifier scores, output length distributions, and formatting validity. This layer is fast, cheap, and catches obvious failures. It does not catch subtle semantic failures or context-dependent quality problems.

The middle layer is LLM-based evaluation on a larger sample of outputs, scoring against defined rubric dimensions. This layer is more expensive per sample than pure automated metrics but cheaper than human evaluation. It covers quality dimensions that simple metrics cannot measure, at a scale that human evaluation cannot match. It requires periodic validation against human evaluation to ensure the LLM judge is behaving as expected.

The top layer is human evaluation on a smaller but carefully constructed sample: high-stakes outputs, examples where the lower layers flagged anomalies, and a random baseline sample for ongoing quality monitoring. This layer provides the ground truth signal that validates the other layers and catches failure modes that automated and LLM-based methods miss.

Maintaining Evaluation Quality Over Time

Human evaluation is not a one-time activity. For evaluation programs that run over months or years, maintaining quality requires ongoing attention to annotator calibration, rubric currency, and measurement of evaluation consistency.

Annotators drift in their interpretations over time. A calibration session from six months ago may not reflect current consensus, especially if the model's outputs have changed in ways that exposed rubric gaps. Periodic recalibration, every month or two for active annotation programs, keeps the team aligned. Including gold-standard examples with known scores in each annotation batch allows ongoing monitoring of whether individual annotators are drifting from calibrated interpretations.

Rubrics also need to evolve. As the model and deployment context change, new failure modes emerge that the original rubric did not address. A rubric that was comprehensive at launch may have significant gaps twelve months later. Reviewing the rubric whenever monitoring finds unexpected output patterns, and updating it to address newly discovered failure modes, keeps the evaluation program relevant.

Handling Disagreement

Not all disagreement is a problem. Genuine ambiguity in output quality is information: if highly qualified annotators disagree about whether an output is good, that may indicate the output is in a genuinely borderline quality range, not that the rubric is broken or the annotators are incompetent. Tracking disagreement rates by output category, by annotator pair, and over time helps distinguish between rubric problems, annotator calibration problems, and genuine ambiguity.

For outputs where annotators systematically disagree, the right response depends on the source of disagreement. If it traces to a rubric ambiguity, fix the rubric and re-score the affected examples. If it traces to an annotator applying criteria idiosyncratically, invest in retraining. If it traces to genuine output ambiguity, consider using the disagreement itself as a signal: outputs that qualified humans disagree about may warrant extra caution in deployment, regardless of the ultimate score assigned.

Human Evaluation in the Context of Agentic AI

As AI systems become more agentic, taking sequences of actions rather than producing single outputs, human evaluation faces new challenges. Evaluating a single response is relatively straightforward. Evaluating whether a multi-step agent completed a task correctly, in an appropriate way, without taking unintended actions along the path, requires evaluation approaches that traditional annotation frameworks were not designed for.

Agentic evaluation requires annotators to assess both the final outcome and the process that led to it. An agent that produces the correct final result through a problematic intermediate step may not be acceptable in deployment, even if an output-only evaluation would score it as successful. Conversely, an agent that takes all the right steps but fails on the last one due to an external failure may deserve more credit than an output-only evaluation would assign.

Human evaluation for agentic systems also requires annotators who understand the action space: what actions was the agent capable of taking, which did it take, and were the actions it took appropriate given the task and context? This requires domain knowledge about both the AI system and the task domain, and it typically requires evaluators to review logged action traces rather than just inputs and outputs. The annotation infrastructure for agentic evaluation is more complex than for single-turn output evaluation, but the principles remain the same: clear rubrics, calibrated annotators, measurable inter-annotator agreement, and feedback loops from evaluation findings to system improvement.

The Future of Human Evaluation

Human evaluation is becoming more sophisticated as the tasks AI systems perform become more complex. Three directions are particularly worth tracking.

Expert-in-the-loop evaluation, where domain experts provide judgment on specialized outputs, is becoming more common in high-stakes domains where general annotators cannot reliably assess quality. The MEDFIT-LLM study (Rao, Jaggi, Naidu, IEEE RMKMATE 2025) demonstrated the importance of clinical expertise in evaluating medical AI outputs. Similar expert involvement is emerging in legal, financial, and scientific AI evaluation programs.

Comparative evaluation, where annotators judge which of two outputs they prefer rather than scoring a single output against a rubric, is increasingly used because it reduces cognitive load and produces more consistent results. Chatbot Arena and similar platforms have demonstrated that comparative evaluation can scale to large volumes while maintaining reliability. The limitation is that preference does not always equal correctness, so comparative evaluation is most appropriate for tasks where quality is inherently comparative rather than absolute.

Longitudinal evaluation, where the same outputs are evaluated at multiple points in time, is gaining attention as a way to measure temporal validity. A response that is accurate today may become inaccurate as facts change, guidelines update, or regulations shift. Longitudinal evaluation programs that re-score stored outputs against current standards provide a continuous measurement of the model's temporal validity that point-in-time evaluation cannot.