Evaluators

Create and manage LLM-as-judge evaluators and their versions programmatically.

The evaluators client methods are currently in BETA. The API may change without notice. A one-time warning is emitted on first use.

Create and manage LLM-as-judge evaluators and their versions programmatically. Evaluators use prompt templates with {variable} placeholders that reference span or trace attributes to automatically score your LLM application’s outputs.

Key Capabilities

• Create template-based LLM-as-judge evaluators within a space
• Version evaluators with commit messages (versions are immutable once created)
• Retrieve evaluators with their latest or a specific version
• List, update, and delete evaluators
• List and retrieve individual evaluator versions

List Evaluators

List all evaluators you have access to, with optional filtering by space.

resp = client.evaluators.list(
    space="your-space-name-or-id",  # optional
    name="Relevance",               # optional substring filter
    limit=50,
)

for evaluator in resp.evaluators:
    print(evaluator.id, evaluator.name)

For details on pagination, field introspection, and data conversion (to dict/JSON/DataFrame), see Response Objects.

Create a Template (LLM-as-Judge) Evaluator

Create a new template evaluator with an initial version. Evaluator names must be unique within the target space.

from arize.evaluators.types import TemplateConfigInput, EvaluatorLlmConfigRequest, OptimizationDirection

evaluator = client.evaluators.create_template_evaluator(
    name="Relevance",
    space="your-space-name-or-id",
    commit_message="Initial version",
    description="Scores whether the response is relevant to the query",
    template_config=TemplateConfigInput(
        name="Relevance",
        template="Is the following response relevant to the query?\nQuery: {input.value}\nResponse: {output.value}",
        include_explanations=True,
        use_function_calling=True,
        classification_choices={"relevant": 1, "irrelevant": 0},
        direction=OptimizationDirection.MAXIMIZE,
        llm_config=EvaluatorLlmConfigRequest(
            ai_integration_id="your-ai-integration-id",
            model_name="gpt-4o",
            invocation_parameters={"temperature": 0},
            provider_parameters={},
        ),
    ),
)

print(evaluator.id, evaluator.name)

Create a Code Evaluator

Create a new code evaluator with an initial version. Use ManagedCodeConfigRequest for built-in checks (JSONParseable, Regex, KeywordMatch, ExactMatch) or CustomCodeConfigRequest for user-supplied Python.

from arize.evaluators.types import ManagedCodeConfigRequest

evaluator = client.evaluators.create_code_evaluator(
    name="JSON Parseable",
    space="your-space-name-or-id",
    commit_message="Initial version",
    code_config=ManagedCodeConfigRequest(
        type="managed",
        name="json_parseable",
        managed_evaluator="JSONParseable",
        variables=["output"],
    ),
)

print(evaluator.id, evaluator.name)

Evaluator name must match the regex ^[a-zA-Z0-9_\s\-&()]+$.

Template Variables

Template strings use {variable} placeholders (f-string format) that reference span or trace attributes (e.g., {input.value}, {output.value}, {attributes.my_custom_attr}).

Classification vs. Freeform Output

• Classification — Provide classification_choices as a dict[str, float] mapping label → numeric score (e.g., {"relevant": 1, "irrelevant": 0}). The evaluator outputs one of these labels along with its score.
• Freeform — Omit classification_choices. The evaluator produces a numeric score without predefined labels.

Get an Evaluator

Retrieve an evaluator by name or ID. By default the latest version is returned. When using a name, provide space to disambiguate.

evaluator = client.evaluators.get(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
)

print(evaluator.id, evaluator.name)
print(evaluator.version)

Get a Specific Version

evaluator = client.evaluators.get(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    version_id="specific-version-id",
)

Update an Evaluator

Update an evaluator’s metadata (name and/or description). To change the template configuration, create a new version instead.

evaluator = client.evaluators.update(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    name="Relevance v2",
    description="Updated description",
)

print(evaluator)

Delete an Evaluator

Delete an evaluator and all its versions. This operation is irreversible. There is no response from this call.

client.evaluators.delete(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
)

print("Evaluator deleted successfully")

Manage Versions

Evaluator versions are immutable once created. To change the template configuration, create a new version — it becomes the latest version immediately.

List Versions

List all versions for an evaluator.

resp = client.evaluators.list_versions(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    limit=50,
)

for version in resp.evaluator_versions:
    print(version.id, version.commit_message)

For details on pagination, field introspection, and data conversion (to dict/JSON/DataFrame), see Response Objects.

Get a Version

Retrieve a specific evaluator version by its ID.

version = client.evaluators.get_version(version_id="your-version-id")

print(version.id, version.commit_message)

Create a New Template Version

Add a new template version to an existing template evaluator. The new version becomes the latest immediately.

from arize.evaluators.types import TemplateConfigInput, EvaluatorLlmConfigRequest, OptimizationDirection

version = client.evaluators.create_template_version(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    commit_message="Improved prompt for edge cases",
    template_config=TemplateConfigInput(
        name="Relevance",
        template="Rate the relevance of the response on a scale of 0 to 1.\nQuery: {input.value}\nResponse: {output.value}",
        include_explanations=True,
        use_function_calling=True,
        classification_choices={"relevant": 1, "irrelevant": 0},
        direction=OptimizationDirection.MAXIMIZE,
        llm_config=EvaluatorLlmConfigRequest(
            ai_integration_id="your-ai-integration-id",
            model_name="gpt-4o",
            invocation_parameters={"temperature": 0},
            provider_parameters={},
        ),
    ),
)

print(version.id)

Create a New Code Version

Add a new code version to an existing code evaluator.

from arize.evaluators.types import ManagedCodeConfigRequest

version = client.evaluators.create_code_version(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    commit_message="Updated managed evaluator",
    code_config=ManagedCodeConfigRequest(
        type="managed",
        name="json_parseable",
        managed_evaluator="JSONParseable",
        variables=["output"],
    ),
)

print(version.id)

Delete Versions

Delete a batch of evaluator versions (1-100 IDs per request). This operation is irreversible. The delete is partial-tolerant: IDs that exist and belong to the evaluator are deleted, and requested IDs that were not deleted are returned in not_deleted_version_ids. Deleting a version pinned to a running online task un-pins that task (falls back to latest).

resp = client.evaluators.delete_versions(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    version_ids=["your-version-id-1", "your-version-id-2"],
)

print(resp.completed)
print(resp.deleted_version_ids)
print(resp.not_deleted_version_ids)

Learn more: Online Evaluations Documentation