LLM features need a test suite, not vibes

There is a scene that plays out in every AI feature team, roughly every two weeks. Someone upgrades the model, or changes the prompt, or tweaks the retrieval, and the demo looks great. The answers are snappier, the formatting is cleaner, everyone is excited. Two days later the support channel fills up with complaints about answers that used to work. Nobody can say what regressed, because nothing was measured. The vibes were good, and the vibes lied.

I have been on both sides of this. I have shipped model changes that made the demo better and the product worse, and I have spent weeks reconstructing what broke from memory, because the traces did not exist. The fix is not exotic. It is the same discipline you apply to any codebase: a test suite, run on every change, that catches regressions. For language model features, that test suite is called an evaluation set, and it is the difference between an AI feature and an AI incident.

Start with a golden set, not a spreadsheet

The first eval set is embarrassingly simple to build, and I do not mean that as a compliment to the people who skip it. Take a hundred realistic inputs, maybe fifty. Write down what a good output looks like for each one. That is it. That is the foundation.

The inputs should come from reality, not imagination. Dig through your logs, your support tickets, your customer emails. Find the questions people actually asked, the documents they actually uploaded, the edge cases that actually hurt. A golden set built from hypothetical questions is a golden set that validates your imagination. The production set validates your product.

Store it next to your code, in version control. The eval set should change with your codebase, and a model upgrade that breaks ten cases in the golden set is exactly the kind of news you want delivered by a red test run, not by a support ticket.

Test the parts, and test the whole

A language model feature is a pipeline, and the pipeline has separate failure surfaces. Retrieval can fetch the wrong context. Generation can misuse the context. Formatting can produce output that does not parse. Each of these needs its own tests, because each has a different fix.

The retrieval layer can be tested without the model at all. Give a query, assert that the expected document is in the top results. This is a pure search metric, deterministic in the sense that you can measure it, and it runs fast. Most teams discover the retrieval is the weak link precisely when they write these tests, because for the first time they see the recall number instead of a convincing answer.

The generation layer needs judgment, which means a human or a strong model acting as judge. Ask the question, get the answer, and score it on the dimensions that matter to you: does it answer the question, does it stay grounded in the provided context, does it follow the format, does it invent information. A rubric makes the judgment reproducible. Without a rubric, your eval results are vibes with a score attached.

The whole pipeline needs end to end tests too, because the parts can be perfect and the combination broken. The user does not experience your retrieval metrics. They experience the final answer.

LLM as judge works, with guardrails

If you have ever seen the sheer volume of outputs a real eval set produces, you know the human review problem: a hundred cases, run across every prompt change, is a lot of reading. The pragmatic answer is a strong model acting as a judge, scoring outputs against the rubric.

This is legitimate, and it is also easy to do badly. The judge model needs to be a different model from the one under test, or it inherits the same blind spots. The rubric needs to be explicit, because an unscored "is this good" prompt produces whatever the judge felt like. And you need to sample human review of the judge's decisions, because the judge is a model too, and models agree with themselves in ways that are not always true.

The failure mode to watch is judge drift. The judge model gets upgraded, or the rubric wording changes, and suddenly every output scores better or worse for reasons that have nothing to do with your feature. Your eval set is only useful if the judge is stable, so pin the judge model and treat rubric changes as events that require revalidation.

The eval set is a regression test suite

The framing that finally made this click for me: the eval set is not a benchmarking exercise, it is a regression test suite for a system that cannot be diffed. You cannot look at a prompt change and reason about what it will do to a hundred unseen inputs. You can only measure. The eval set is how you measure, and running it on every candidate change is how you prevent regressions.

That means the evals need to run in the loop, not as an occasional audit. In CI, gated on the changes you care about. The fast parts, retrieval tests and format checks, run on every commit. The expensive parts, judge scored generation evals, run on every model or prompt change. This is where the discipline lives. A team that runs evals once a quarter has a report. A team that runs them on every change has a quality floor.

Measure the direction, not just the average

The temptation is to roll everything up into a single score. The score goes up, you are good, the score goes down, you are bad. That hides everything that matters. A prompt change that improves ninety five cases and wrecks five is a prompt change that will wreck five real users.

Slice the results. Group by input type, by domain, by difficulty, by language. A change is good if it improves the cases you care about without damaging the ones you already handled. The aggregate hides the trade, and the whole point of evals is to see the trade before your users do.

The honest starting point

You do not need a fancy eval platform to start. You need a hundred real inputs, a rubric, and a script that runs the pipeline and scores it. That is days of work, not months, and it pays for itself the first time it stops you from shipping a regression.

The teams that treat model quality as a feeling will keep having quality problems. The teams that treat it as a test suite will keep having a quality floor. Every prompt engineer I respect is really an evaluation engineer with a nicer title.