06 · Evaluation and experiments
Evaluating generated answers
9 min read
A generated answer, such as a summary, a chat reply or a written response to a search, has no single right answer to compare against. The counting methods of the previous two topics therefore do not apply. Generative evaluation is the set of ways to grade such outputs anyway, using people, reference answers and other models as judges. The design decision is how much of the grading budget goes to each, and which checks are automatic enough to run on every release.
An everyday example
A spelling test is marked against an answer key, and two teachers marking the same paper agree to the letter. An essay has no key. A teacher marks it against a rubric, which is a short list of what a good essay does, such as answering the question asked, getting the facts right and staying within the length. A second teacher marking the same essay will usually agree within a grade, but not always.
A school with a thousand essays and two teachers has a budget problem. It can have every essay double-marked and wait a month, mark a sample and trust the rest, or train senior students to mark with the rubric and check their marks against the teachers now and then. Generated answers are the essays, the rubric is the grading scheme, and the senior students are the judge models described further down.
On volume and on honesty the essay stops matching. A system writes a hundred thousand answers a day, so even sampling is a large job, and some answers are confidently wrong in ways a quick read will not catch, while a student essay is rarely a convincing fabrication.
Write down what a good answer does
Grading starts with a rubric, and a rubric is a short list of qualities each stated so that two people can check it the same way. For a support chat the qualities might be that the answer resolves the question asked, that every fact in it appears in the company’s documents, that it promises nothing the policy forbids, and that it is under 120 words. A quality that two people cannot check independently, such as sounding friendly, is replaced by one they can, such as using no blame language.
The rubric is applied to a golden set, which is a fixed collection of a few hundred inputs with human grades for the outputs, chosen to cover the common cases and the known hard ones. Every change to the system is run against the golden set and the grades are compared with the last version. Three hundred examples is a working size. Grading them twice by hand costs about 20 hours at two minutes a grade, and a win rate measured on 300 examples has a margin of error of about six percentage points either way.
Pass or fail per quality beats a one-to-ten score. Two graders agree far more about whether a fact appears in the source than about whether an answer deserves a seven, and a failed quality names the fix.
Pay people for the judgments that matter
People are the most trusted judges and the most expensive. At two minutes per answer and two graders per answer for agreement, a thousand answers cost about 67 hours. Human grading therefore goes to the golden set, to a weekly sample of live traffic and to any case a cheaper judge flagged as uncertain. Agreement between graders is measured too, because a rubric that two trained people apply differently is measuring the graders.
Comparing two answers side by side is easier and more consistent than scoring one. The usual human task is therefore a pair, the old system’s answer and the new one, with the order shuffled so the grader cannot tell which is which. The result is a win rate, the share of pairs where the new answer won. A new version that wins 55 percent of 300 pairs is inside the six-point margin of error and has not been shown to be better.
Live users are graders too, in a weaker way. A thumbs up, a follow-up question that repeats the first one, or a person who gives up and calls support are all signals, and they are cheap and plentiful. They are also biased toward the people who bother to react, so they are read alongside the golden set and never instead of it.
Use a model as a judge, and check the judge
A judge model is a large language model, a model that reads and writes text, given the rubric, the input, the answer and sometimes the source documents, and asked to grade. Assume it costs about a cent an answer, so a thousand answers cost ten dollars instead of 67 hours, and it can run on every release and on a slice of live traffic every hour. That is the reason every generative system uses one.
A judge model has blind spots that are now well documented. It tends to prefer longer answers, and it prefers the first answer in a pair unless the order is shuffled. It prefers answers in its own style and may rate its own family of models higher, and it can be fooled by a confident tone wrapped around a wrong fact. It is also only as good as the rubric it was handed, and a vague rubric gives a vague judge.
The fix is to treat the judge as a model with its own evaluation. Grade the golden set with people, grade it with the judge, and report how often they agree. The target is agreement at least as high as two people have with each other, which for a clear pass-or-fail rubric is around 85 to 90 percent. When agreement drops, the judge’s instructions, called its prompt, are fixed before anything it graded is trusted, and the hardest and most uncertain cases are routed to people.
Reference comparisons and checks that need no judge
When a reference answer exists, such as a human-written summary, the output can be compared with it by word overlap, with scores such as ROUGE for summaries. Overlap is cheap and catches an answer that wandered off topic, and it misses a correct answer that used different words, so it is a screen and never the final grade.
For a system that answers from retrieved documents, the most useful automatic check is grounding, which asks whether each claim in the answer can be found in the documents the system was given. A judge model does this well because the task is close to a reading test with the answer key present. Questions with a checkable answer, such as a date or a price, get exact-match checks, and format rules such as length, language and required fields are plain code.
Every automatic check runs on every release, and a release that lowers grounding or raises the fail rate on the golden set does not ship whatever the win rate says. Live monitoring runs the same checks on a sample of traffic, because the questions people ask change faster than any golden set.
| Check | What it catches | What it misses |
|---|---|---|
| Word overlap with a reference | Summaries that drift off topic | A correct answer worded differently |
| Grounding against sources | Facts that are not in the documents | A true fact the documents left out |
| Exact match on known answers | Wrong dates, names and numbers | Everything that is not a fact |
| Format and length rules | Too long, wrong language, missing fields | Whether the content is right |
Where people get confused
People look for a single accuracy number. A generated answer can be fluent, grounded and useless, or terse, correct and rude, and the rubric exists so that those are separate grades instead of one that hides them.
A judge model also gets trusted because it is a model. It is a grader with known habits, and its grades mean something only after they have been compared with people on the same examples.
The golden set gets skipped because live feedback is free. Thumbs come from the people who react, which is a different population from the people who left, and a fixed set is the only way to tell whether a change made the system better or made the askers different.
In the interview
On a generative problem such as a support assistant or a search that writes answers, evaluation moves forward and takes six to eight minutes, because the interviewer knows there is no accuracy to quote. Open with the rubric in three or four qualities. Say that a golden set of a few hundred human-graded examples is the fixed test, and that a judge model runs the rubric on every release and is checked against people for agreement.
Name grounding as the one automatic check you would never skip, and give the budget arithmetic for human grading, because it shows you know why the judge exists.
Interview questions
How would you evaluate a chatbot that answers questions from our help centre?
I would write a rubric of four pass-or-fail qualities. The answer has to resolve the question, every claim has to appear in the help centre pages the system retrieved, it may promise nothing policy forbids, and it has to stay under a length limit. Then I would build a golden set of about 300 questions, covering the common ones and the known hard ones, and have two people grade the answers so I know how well people agree.
Every release runs against that set with a judge model applying the same rubric, and I would report the judge’s agreement with the human grades, aiming for about 85 percent or better. Grounding is the check I would never skip, because an answer that invents a refund policy is the failure that costs the most.
Can you use a language model to grade its own outputs?
Yes, with a check attached. A judge model can run the rubric on a thousand answers for about ten dollars, where two people would need 67 hours, so it is the only way to grade every release and a slice of live traffic.
It has known habits, such as preferring longer answers, preferring the first of a pair and favouring its own style. I would shuffle pair order, keep the rubric pass-or-fail, and measure its agreement with human grades on the golden set. If agreement falls below what two people manage with each other, the judge gets fixed before its grades are used for anything.
Users already give thumbs up and thumbs down. Why do you need anything else?
Thumbs are useful and I would keep them, but they come from the small share of people who bother to react, and that group changes when the system changes. A release that made answers worse for quiet users and better for vocal ones would look like a win.
A fixed golden set asks the same 300 questions of every version, so a change in its grades is a change in the system and nothing else. I would read the thumbs as a live alarm and the golden set as the measurement.
What to practice next
Design these systems on the canvas and get your design graded.
Cohen’s kappa measures how well two graders agree beyond chance, which is the check this topic applies to people and to judge models, and the bootstrap gives the uncertainty interval around a win rate measured on a few hundred pairs.