Create and validate evaluators

Interpret traces, define success criteria, and build code and LLM-as-a-judge evaluators calibrated against human judgment.

Evaluation turns raw traces into measurable judgments of quality. Interpret what your traces reveal, define the criteria that separate a good response from a bad one, and build evaluators, both code-based and LLM-as-a-judge, whose scores you can rely on.

Before you write evals for agents, read your data | Ep. 5

Read what your traces reveal and establish the criteria that distinguish a good response from a bad one.

Your first code eval for agents: catch bugs in 5 lines of Python | Ep. 6

Score outputs with deterministic checks for criteria that don’t require a model to judge.

LLM-as-a-judge for agents: build a custom eval rubric that works | Ep. 7

Use an LLM to assess qualities that are difficult to capture in code, and write judge prompts that produce consistent scores.

5 LLM and agent eval mistakes that turn metrics into noise | Ep. 8

The common mistakes that make evaluations misleading, and how to avoid them.

Is your LLM judge right? Calibrate with meta-evaluation | Ep. 9

Calibrate an LLM judge against human judgment so you can rely on its scores.

Up next

Experiment, monitor, and export traces

Prove changes with experiments, evaluate production traffic, and turn failures into fixes.