Create an evaluator in Arize AX, attach it to a task on your live project, and filter your traces to the responses it scores as failures.
In Observe your agent, you found a failure by reading one trace. Now catch that failure across ongoing production traffic with an online eval: an evaluation that scores every trace as it arrives.
The judge calls a model, so add a provider first under Settings → Account Settings → Integrations. See AI provider integrations.
The Evaluator Hub, with quickstart templates and score distributions for evaluators already running.
1
Pick a template
Go to Evaluators → New Evaluator and pick the template that matches your failure. This guide uses Hallucination. For objective checks, like an empty response, use a code evaluator.
Templates grouped by what they check, with the variables each one needs.
2
Configure the judge
Name it, set its Scope, pick the model, and adjust the prompt and Choices to your standards. Leave Explanations on so each score says why it passed or failed.
3
Point it at your traces
Set Pick a Data Source to your project, use Filter Spans to choose what’s scored, and use Map Variable to Data to connect {input}, {context}, and {output} to your span attributes. Test Evaluator On Spans tries it on a sample.
Configuring a hallucination evaluator and pointing it at a project.
4
Turn it on
Click Create and Run Evaluator. This creates a task that scores new traffic from now on. Manage it under Running Eval Tasks.
