Evals

An eval is a test for AI output. Rather than eyeballing a few answers and calling it good, you collect a set of real examples, run your prompt or agent over all of them, and score the results, so "it seems fine" becomes a number you can compare. Change the prompt, rerun the eval, and you can see whether it got better or worse instead of guessing.

Scoring is the interesting part, because AI output is rarely a clean right or wrong. Some checks are automatic, like whether the JSON parses or the answer matches a known value. When there's no clean answer to check against, you can have a model grade the output against a rubric you write, which people call LLM-as-judge. Serious teams keep a fixed set of test cases and watch the score move every time they change something.

The moment you're shipping something built on AI, and "it worked when I tried it" isn't good enough. If you're tweaking prompts and can't tell whether your changes help, that's the signal you need an eval, even a rough one. A dozen real examples in a spreadsheet with a pass/fail column beats a hundred vibes.

Skip the ceremony for one-off personal use, where you're checking every answer yourself anyway. It's the repeated work that someone relies on where evals earn their place. And public benchmark scores only describe a model in general, so the eval worth building is usually the one you make from your own examples.