Every team that ships a language model feature eventually reaches the same problem. Someone changes a prompt, everyone agrees the output looks better, it goes out, and something that used to work stops working. Nobody can prove it, because there was never a baseline.
An evaluation set fixes that. It is not sophisticated and it is the difference between engineering and guessing.
Build it from real cases
Constructed examples can test boundaries and attacks that have not appeared in ordinary work. They should complement representative real cases, not replace them. Keep their origin visible so the team knows what the set actually represents.
Collect from your actual work. Support tickets. Real enquiry text. Actual documents. The questions people asked in the first two weeks of a pilot. Include the awkward ones: the message with three questions in it, the document with a table, the enquiry written in a second language, the request that should be refused.
Thirty to fifty cases is enough to be useful. Two hundred is better and is not necessary to start.
Record the expected outcome, not the expected wording
The mistake that makes an evaluation set useless is recording a model output as the correct answer. The next model phrases it differently and scores zero while being just as good.
Record what must be true of the answer instead. For a classification, the correct label. For an extraction, the exact values that should be pulled from the source. For a retrieval grounded answer, the passage that should have been cited and the specific facts the answer must contain. For a refusal, that it refused and offered the handover.
That makes scoring largely mechanical. Did it produce the right label. Did it extract the right figure. Did it cite the right document. Did it refuse.
Measure retrieval separately
This is the highest value distinction in the whole exercise.
For anything grounded in your documents, check first whether the correct passage appeared in the retrieved set. If it did not, no prompt change will produce a correct answer, and a team that does not measure this separately will spend weeks tuning a prompt for a retrieval problem.
Report two numbers: how often the right passage was retrieved, and how often the answer was correct given that it was. The first points at chunking, metadata and search. The second points at the prompt and the model.
Scoring without a human in the loop
Classification and extraction score exactly. Compare to the expected value.
Open ended answers are harder. The workable approaches are checking for required facts as substrings or through a small extraction step, checking that the citation matches the expected source, and using a separate model as a judge with a specific rubric rather than a general quality question.
The judge approach works reasonably when the rubric asks concrete questions. It is unreliable when asked whether an answer is good.
Keep a small subset for human review regardless. Twenty cases read by a person each release catches the kind of degradation no automated score notices, such as an answer that is technically correct and unusable in tone.
Run it as a gate
The evaluation belongs in the pipeline, not in someone notebook.
Any change to a prompt, a model, a retrieval parameter or a chunking strategy runs the set and reports the scores against the previous run. A drop beyond a threshold blocks the change.
This makes regressions visible against an agreed baseline. It does not establish safety for every unseen input, so preserve monitoring and a rollback path after release.
Grow it from failures
A reported wrong answer should become a reviewed case with an expected outcome. Repeated runs can still vary, so record that variability rather than assuming one passing run prevents every recurrence.
Over a year the set becomes a record of everything the system has got wrong, which is the most valuable test asset in the product.
Measure cost and latency too
The same run should report tokens consumed and response time per case. A prompt change that improves accuracy by two percent and doubles the cost is a decision, and you cannot make it without both numbers.
For interactive features, latency is a quality attribute. Record whether the answer arrives in time for the task, not just whether its final wording is correct.
Keep a holdout set separate from prompt editing
If engineers repeatedly adjust a prompt against the same examples, the prompt can improve on those examples without improving on new work. Keep a separate set that is not used for routine editing, and run it when selecting a release candidate. Record which examples influenced the change so the distinction remains meaningful.
Break down results by failure type. An extraction system might perform well on typed forms and poorly on photographed pages. A single average can hide that difference. Report missing fields, incorrect values, unsupported additions and appropriate abstentions separately. The acceptable tradeoff depends on what happens to the output next.
Check the evaluator itself. A substring match can pass an answer that quotes the correct number and then recommends the wrong action. A model judge can be inconsistent or favour a particular writing style. Review disagreements against the source material and revise the rubric when it rewards the wrong behaviour.
Keep the test inputs private where they contain business information. Use approved, minimised examples, restrict access and remove identifiers that the task does not require. The evaluation pipeline must follow the same data boundary as the feature being tested; sending a private deployment's test cases to an unrelated hosted judge would defeat that boundary.
Publish the release decision with the model version, prompt version, retrieval configuration and known weak categories. That record lets the team investigate a later change without relying on memory or a screenshot of one convincing answer.