How do I know if my AI feature or prompt is any good?

An eval is a test for AI output. You collect real examples, run your prompt, skill, or agent over all of them, and score the results, so "it seems fine" becomes a number you can compare. Change the prompt, rerun, and you see whether it got better or worse. How you score depends on whether the task has a checkable answer: - Bounded tasks get automatic checks. Does the JSON parse, does the code follow the convention, did the security check pass, does the answer match a known value. These run in seconds. - Open-ended work gets a judge. A feature mockup or a sales email has no single right answer, so a person or a second model grades it against a rubric you write. Using a model this way is called LLM-as-judge, the same idea as a critique loop. A confident answer can still be a hallucination, so the rubric should check facts as well as tone.

I changed a prompt and I can't tell if it got better or worse.

An eval is a test for AI output. You collect real examples, run your prompt, skill, or agent over all of them, and score the results, so "it seems fine" becomes a number you can compare. Change the prompt, rerun, and you see whether it got better or worse. How you score depends on whether the task has a checkable answer: - Bounded tasks get automatic checks. Does the JSON parse, does the code follow the convention, did the security check pass, does the answer match a known value. These run in seconds. - Open-ended work gets a judge. A feature mockup or a sales email has no single right answer, so a person or a second model grades it against a rubric you write. Using a model this way is called LLM-as-judge, the same idea as a critique loop. A confident answer can still be a hallucination, so the rubric should check facts as well as tone.

How do I test whether my new skill is better than the old one?

An eval is a test for AI output. You collect real examples, run your prompt, skill, or agent over all of them, and score the results, so "it seems fine" becomes a number you can compare. Change the prompt, rerun, and you see whether it got better or worse. How you score depends on whether the task has a checkable answer: - Bounded tasks get automatic checks. Does the JSON parse, does the code follow the convention, did the security check pass, does the answer match a known value. These run in seconds. - Open-ended work gets a judge. A feature mockup or a sales email has no single right answer, so a person or a second model grades it against a rubric you write. Using a model this way is called LLM-as-judge, the same idea as a critique loop. A confident answer can still be a hallucination, so the rubric should check facts as well as tone.

The demo works, but how do I know it holds up when it's not just me trying it?

An eval is a test for AI output. You collect real examples, run your prompt, skill, or agent over all of them, and score the results, so "it seems fine" becomes a number you can compare. Change the prompt, rerun, and you see whether it got better or worse. How you score depends on whether the task has a checkable answer: - Bounded tasks get automatic checks. Does the JSON parse, does the code follow the convention, did the security check pass, does the answer match a known value. These run in seconds. - Open-ended work gets a judge. A feature mockup or a sales email has no single right answer, so a person or a second model grades it against a rubric you write. Using a model this way is called LLM-as-judge, the same idea as a critique loop. A confident answer can still be a hallucination, so the rubric should check facts as well as tone.

What are "evals" and why does everyone building with AI keep talking about them?

An eval is a test for AI output. You collect real examples, run your prompt, skill, or agent over all of them, and score the results, so "it seems fine" becomes a number you can compare. Change the prompt, rerun, and you see whether it got better or worse. How you score depends on whether the task has a checkable answer: - Bounded tasks get automatic checks. Does the JSON parse, does the code follow the convention, did the security check pass, does the answer match a known value. These run in seconds. - Open-ended work gets a judge. A feature mockup or a sales email has no single right answer, so a person or a second model grades it against a rubric you write. Using a model this way is called LLM-as-judge, the same idea as a critique loop. A confident answer can still be a hallucination, so the rubric should check facts as well as tone.

Concepts

Evals

Updated 2026-10-11

Questions this answers

How do I know if my AI feature or prompt is any good?
I changed a prompt and I can't tell if it got better or worse.
How do I test whether my new skill is better than the old one?
The demo works, but how do I know it holds up when it's not just me trying it?
What are "evals" and why does everyone building with AI keep talking about them?

The Fix

An eval is a test for AI output. You collect real examples, run your prompt, skill, or agent over all of them, and score the results, so "it seems fine" becomes a number you can compare. Change the prompt, rerun, and you see whether it got better or worse.

How you score depends on whether the task has a checkable answer:

Bounded tasks get automatic checks. Does the JSON parse, does the code follow the convention, did the security check pass, does the answer match a known value. These run in seconds.
Open-ended work gets a judge. A feature mockup or a sales email has no single right answer, so a person or a second model grades it against a rubric you write. Using a model this way is called LLM-as-judge, the same idea as a critique loop.

A confident answer can still be a hallucination, so the rubric should check facts as well as tone.

When to Use It

As soon as other people rely on the output, or you're tweaking prompts and can't tell whether the changes help. A dozen real examples in a spreadsheet with a pass/fail column is enough to start.

The smallest eval we teach is a side-by-side run. Before you replace a skill, run the old and new versions on the same three tasks, then read the outputs next to each other. Claude Code can run both without supervision in headless mode (claude -p), so the comparison takes minutes. If the new one isn't clearly better on all three, keep the old one.

Public benchmark scores describe a model in general. The eval worth building is the one made from your own examples. Writing and testing skills this way is part of our AI training for teams.