Module 3: Human Evaluation of AI · Evaluating AI Models

A language model scores well on BLEU for a translation task (illustrative). Native speakers rate the same translations poorly for naturalness (illustrative). The automated metric and the human experience of the output are pointing in opposite directions. This module teaches you how to design a human evaluation study that captures what automated metrics miss: fluency, tone, cultural fit, and whether the output is actually useful.

What Automated Metrics Miss

Automated metrics measure the properties of text that can be computed without understanding meaning. BLEU counts n-gram overlap. ROUGE counts recall of reference n-grams. Perplexity measures how predictable the output is given a language model. These calculations are fast, consistent, and reproducible.

They miss several things that humans notice immediately. Fluency is the overall feel of a text as natural language. A text can have high n-gram overlap with a reference and still read awkwardly. Tone matters for outputs that will be shown to users: a technically correct medical explanation written in clinical jargon is less useful than one written at an accessible reading level. Cultural fit is almost entirely absent from automated metrics: a translation that uses the correct words in the wrong register, or omits an idiom that would be natural in the target language, will score normally on BLEU but be rated poorly by native speakers. Coherence across a long output is another gap: automated metrics applied segment-by-segment do not detect whether the full output has internal consistency and a clear logical flow.

For any use case where the output is shown to real users, a human evaluation study is not optional. It is the only way to measure whether the model is actually producing outputs that users find helpful, clear, and appropriate.

Designing the Human Evaluation Study

A human evaluation study has five design decisions, each of which significantly affects whether the results are trustworthy.

Cohen's Kappa: Measuring Annotator Agreement

Inter-annotator agreement measures whether multiple annotators produce similar ratings on the same items. If two annotators rate the same 100 summaries and agree on 80 of them, the raw agreement rate is 80%. But this overstates real agreement because some fraction of the agreements happened by chance, especially on scales with fewer categories.

Cohen's kappa corrects for chance agreement. It computes the observed agreement minus the expected chance agreement, divided by the maximum possible agreement above chance. A kappa of 0 means agreement is no better than chance. A kappa of 1 means perfect agreement. Negative kappa means agreement is worse than chance, which usually indicates a problem with the task instructions.

Typical interpretation ranges: kappa below 0.20 is poor; 0.21-0.40 is fair; 0.41-0.60 is moderate; 0.61-0.80 is substantial; above 0.80 is nearly perfect. For AI evaluation studies, a threshold in the substantial agreement range is generally considered the minimum for trustworthy aggregate results. Studies reporting results below the moderate threshold should be interpreted cautiously.

What low kappa usually means When kappa is below 0.40, the most common causes are: the task instructions are ambiguous, the rating scale has too many categories for the annotators to distinguish, the dimension being judged is genuinely subjective (tone, style, creativity), or the annotators are not sufficiently qualified. Fix the instructions first. If kappa is still low after fixing instructions, the dimension may require expert annotators or a binary (yes/no) scale instead of a multi-point Likert scale.

Comparative Evaluation: The Chatbot Arena Approach

Chatbot Arena (Zheng et al., arXiv:2306.05685) introduced a systematic approach to comparative human evaluation that has become widely influential. Instead of asking annotators to rate individual model outputs on a scale, it asks them which of two outputs they prefer for the same prompt.

This format has several advantages. Relative judgments are consistently easier and more reliable than absolute ratings. The pairwise format avoids scale calibration differences between annotators. By collecting thousands of pairwise comparisons from real users interacting with real models, Chatbot Arena produces rankings that better reflect actual user preference than lab-based Likert ratings.

The limitation is cost: collecting enough pairwise comparisons for statistical reliability requires many annotations. For internal evaluation at smaller scale, a modified version works well: present 50-200 prompt-output pairs from each model, ask annotators to choose which response they would prefer to receive if they had asked the question, and compute win rates with confidence intervals.

Annotation fatigue is a real problem in human evaluation studies. Annotators rating many items in a single session show declining quality. Rating accuracy drops, response variance increases, and annotators begin applying shortcuts. The practical solution is to cap session length to a manageable duration, include attention check items (obvious correct answers) to detect fatigue, and rotate annotators across long studies rather than having the same person rate everything.

Python · Cohen's Kappa Calculation

# Cohen's kappa for inter-annotator agreement
# Works for two annotators on categorical or ordinal ratings

from collections import Counter
from typing import List

def cohens_kappa(ratings_a: List[int], ratings_b: List[int]) -> float:
    """
    Calculate Cohen's kappa for two annotators.
    ratings_a, ratings_b: lists of integer ratings (same length)
    Returns kappa in range [-1, 1]
    """
    assert len(ratings_a) == len(ratings_b), "Lists must be the same length"
    n = len(ratings_a)

    # Observed agreement: fraction where both annotators agree
    observed = sum(a == b for a, b in zip(ratings_a, ratings_b)) / n

    # Expected chance agreement: sum of (P_a(category) * P_b(category))
    categories = set(ratings_a) | set(ratings_b)
    count_a = Counter(ratings_a)
    count_b = Counter(ratings_b)
    expected = sum((count_a[c] / n) * (count_b[c] / n) for c in categories)

    if expected == 1.0:
        return 1.0  # Perfect agreement by chance — trivial case

    kappa = (observed - expected) / (1 - expected)
    return round(kappa, 4)


def interpret_kappa(kappa: float) -> str:
    if kappa < 0:      return "Worse than chance — check instructions"
    elif kappa < 0.20: return "Poor agreement"
    elif kappa < 0.40: return "Fair agreement"
    elif kappa < 0.60: return "Moderate agreement"
    elif kappa < 0.80: return "Substantial agreement"
    else:              return "Near-perfect agreement"


# Example: two annotators rating helpfulness 1-5
annotator_a = [4, 3, 5, 2, 4, 3, 5, 4, 3, 2]
annotator_b = [4, 3, 4, 2, 5, 3, 5, 4, 3, 3]

k = cohens_kappa(annotator_a, annotator_b)
print(f"Kappa: {k}")
print(f"Interpretation: {interpret_kappa(k)}")
# Kappa: 0.7027
# Interpretation: Substantial agreement

Try this

Pick a specific AI output type your team evaluates or cares about — summaries, draft emails, code explanations, customer responses. Write one well-formed evaluation question for that output type using the format: "On a scale of 1 to 5, how well does this [output type] [specific criterion], where 1 means [clear low-end description] and 5 means [clear high-end description]?" Then give the same three example outputs to two colleagues and ask each to rate independently using your scale. Compute the agreement rate. The result tells you whether your rating task is clear enough to be reliable.

Human Evaluation

What Automated Metrics Miss

Designing the Human Evaluation Study

Cohen's Kappa: Measuring Annotator Agreement

Comparative Evaluation: The Chatbot Arena Approach

References