Build a dataset

Turn traces, reviewer feedback, or your own examples into a dataset you can reuse to compare prompt, model, and pipeline changes

In Arize AX, a dataset is the fixed set of examples you rerun in experiments to compare changes to your app over time. It gives you a stable benchmark, so you can tell whether a prompt, model, or pipeline update actually improved results or introduced regressions.

Datasets page in Arize AX

Experiment planning

An experiment runs a fixed dataset through one version of your app and scores the results. The experiment type determines what the dataset must contain, so choose it before you build rows.

From any dataset, New Experiment offers four paths:

Path	Use it when	Code required
Run in Agent Playground	The change is in an agent you have already deployed behind an HTTP endpoint.	None in Arize AX, but the agent must be reachable over HTTPS
Run via Code	The task is a function you can call from your own process, including pipelines, sandboxes, and anything needing runtime credentials.	Python, TypeScript, CLI, or REST
Run via Agent (Skills)	You want your AI coding agent to write and run the experiment through the Arize skills.	Driven by your coding agent
Run in Prompt Playground	The change fits inside a single prompt call: wording, model, parameters, or tool definitions.	None

Experiment dataset requirements

Each experiment type reads dataset rows differently, which changes how you name columns.

Experiment type	How it reads the dataset	What the dataset needs
Agent experiment	Each {{dataset.column}} placeholder in the request body is filled from the current row, and the row is sent to your agent’s endpoint.	A column per placeholder, named exactly as referenced.
Code experiments	Your task function receives the whole row and reads whichever fields it needs.	Whatever fields your task function reads.
Prompt playground	Each {variable} in the prompt is filled from a column of the current row.	A column per prompt variable.
Evaluators on any of the above	Evaluators compare the output against reference columns in the row.	An expected output or label column, when the evaluator needs one.

• Column names are part of the contract. Placeholders and prompt variables resolve by exact column name, so renaming a column breaks every run that references it. Keep names stable across dataset versions. Names you choose read best as variables, such as question or expected_category. Trace-sourced columns keep their OpenInference names and work as placeholders unchanged, so attributes.input.value becomes {{dataset.attributes.input.value}}.
• A dataset built for one experiment type usually works for another. The same rows that fill a prompt variable can fill a field in an agent request body. A regression set collected while testing prompts stays useful once that logic moves into a deployed agent.

Choose the variable

Build the dataset around a real failure rather than a hunch. The failing rows become both the variable you test and the benchmark that proves the fix.

If you do not have traces in Arize AX yet, set up tracing first so you have real failures to work from.

1. Open a bad outcome in Traces: a thumbs-down, a low eval score, an exception, or anything a reviewer flagged.
2. Step through the trace span by span. Was the prompt ambiguous? Did the model ignore an instruction? Did the retriever return the wrong chunk? Did a tool call fail to parse?
3. Identify the step that most directly caused the bad outcome. That step is your experimental variable, and the spans around it are the rows worth collecting.

Test one variable at a time. If you change the prompt, the model, and the retriever together, a score change tells you nothing about which of the three caused it. Split them into separate experiments against the same dataset and evaluators.

Common variables to test:

• Prompt: wording, few-shot examples, output schema, or tool definitions.
• Model: the same prompt against a different LLM.
• Invocation parameters: temperature, top-p, max tokens.
• Pipeline or agent behavior: retrieval strategy, routing, tool-using loops, or any logic that lives outside a single prompt.

Plan a baseline

Your first run against a new dataset is the unchanged setup: the prompt, model, and parameters you use today. Name that run baseline-original-prompt. Name each variant after the variable under test, such as variant-concise-prompt or variant-gpt-5.5, so the comparison view stays readable once you have a dozen runs on the same dataset.

If your task predicts labels, decide now which column holds ground truth, which holds the prediction, and which label counts as the positive class. You need all three for the classification metrics setup.

Dataset design

Two things determine whether a dataset is useful: which rows it holds, and how those rows are shaped. The experiment type constrains both.

What to include

A useful dataset blends typical examples that represent everyday traffic, edge cases the app has struggled with (ambiguous inputs, long contexts, unusual formats), and known failures pulled from traces, evaluator results, or reviewer feedback. Without typical examples, you optimize for edge cases and regress on the common path; without failures, you can’t prove a fix actually holds.

Dataset types

You’ll also see datasets described by their source or purpose. These labels overlap and shift as a dataset matures:

• Regression: Examples where the app has already failed. Use these to verify a fix holds and doesn’t quietly reintroduce the bug.
• Golden: Inputs with hand-labeled expected outputs, a stable benchmark for comparing prompt and model changes.
• Synthetic: Generated examples that mimic real inputs. Useful when production data is thin, sensitive, or missing the edge cases you want to stress-test.

A regression set becomes part of a golden dataset once you label the expected output for each row. Collect failures first, label them as you go, and fold in typical traffic so the benchmark isn’t just past bugs.

Dataset row schema

Each row can include input messages, expected outputs, metadata, or any other columns your task function needs. Trace-sourced rows follow the OpenInference convention (e.g., attributes.input.value). CSVs and inline examples use your own column names. Keep them consistent across sources.

Common dataset row shapes

The labels above describe why a row belongs in the dataset. The row itself should match what your task function reads. Common patterns include:

Key-value rows. Use this when the task needs multiple fields such as an input, retrieved context, and an expected output.

Input	Context	Output
What is Paul Graham known for?	Paul Graham is an investor, entrepreneur, and computer scientist known for...	Paul Graham is known for co-founding Y Combinator...

Prompt-completion pairs. Use this for the simplest single-turn completion or classification cases.

Input	Output
"do you have to have two license plates in ontario"	"True"

Messages or chat rows. Use this when your task expects multi-message inputs or outputs.

{
  "input": {
    "messages": [{"role": "system", "content": "You are an expert SQL assistant"}]
  },
  "output": {
    "messages": [{"role": "assistant", "content": "SELECT * FROM users;"}]
  }
}

Choose the shape that matches your task function and keep it consistent within a dataset version.

Create a dataset

Pick the tool you work in. Each tab covers the trace-based, file-upload, and synthetic paths where they apply.

By Arize Skills

The Arize skills plugin wires dataset and trace workflows into your coding agent through the ax CLI.

From traces. Combine arize-trace with arize-dataset. Try:

• “Export error spans from the last 7 days in my production-chatbot project and create a dataset called error-regression-v1.”
• “Find spans where annotation.hallucination.label = 'yes' over the past 14 days and save them as hallucination-examples.”

From a local file. Point the arize-dataset skill at a CSV, JSON, JSONL, or Parquet file you already have. Try:

• “Create a dataset called billing-qa-v1 from ./data/billing_qa.csv in my support space.”
• “Append the rows in new_edge_cases.jsonl to my existing edge-cases dataset.”

Generate synthetic rows. Have the agent draft examples for you. Try:

• “Generate 50 synthetic billing support tickets with query and expected_category fields, then save as support-synthetic-v1.”
• “Draft 20 adversarial inputs targeting prompt injection for my chat agent and save as adversarial-v1.”

Running Arize skills from your coding agent.

By Alyx

Alyx builds datasets directly from the app. It’s available on the pages where you’re already looking at traces, datasets, and prompts (see Alyx meets you where you are for the full list of surfaces). Ask Alyx to turn specific spans into dataset rows, or have it draft synthetic examples when you don’t have production data to pull from.

From traces. Try:

• “Add this span and every similar error to a new regression dataset.”
• “Show me the most common failure patterns in the last 24 hours and add one example of each to a new dataset.”

Generate synthetic rows. Try:

• “Create a synthetic dataset of 100 examples covering billing, technical, and general support categories.”
• “Generate 30 edge cases for my router prompt and save them as a new dataset.”

Ask Alyx to create a new dataset from the pages where you already work.

By UI

From the Traces table. Filter by status, eval score, latency, annotations, or a natural-language query via AI Search. For example, status_code = 'ERROR' for exceptions, eval.groundedness.score < 0.5 for low-scoring spans, or “traces with hallucinations from yesterday”.

Filter the Traces table to find the spans you want.

Select the spans you want and click Add to Dataset to create a new dataset or append to an existing one. Map at minimum the span’s input and output (stored under attributes.input.value and attributes.output.value); for classification tasks, also include a column with the expected label.

Select spans in the Traces table and add them to a dataset.

Upload a file. Go to Datasets & Experiments, click New Dataset, then choose Upload CSV and select a file you generated elsewhere.

Upload a CSV in the New Dataset dialog.

Start with no datasets yet. Before your space contains a dataset, the Datasets & Experiments page offers three shortcuts. Start with Alyx generates a starter dataset, Create regression dataset from traces turns low-scoring interactions into a regression set, and Upload a dataset takes a CSV you already have.

Add rows to an existing dataset. On a dataset, click New Examples to add rows: Add Manually for a row you type in, Select Traces to pull in spans from a project, Upload CSV, Add via Code, or Add via Agent (Skills) to work through the Arize skills in your coding agent.

By Code

Use the Arize SDK to create datasets programmatically. For the Python examples, install arize>=8.0.0 and set ARIZE_API_KEY and ARIZE_SPACE_ID in your environment.

Python

import os
from datetime import datetime, timedelta
from arize import ArizeClient

client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
space = os.environ["ARIZE_SPACE_ID"]

# From inline examples
client.datasets.create(
    name="support-qa-v1",
    space=space,
    examples=[
        {"input": "How do I cancel?", "expected_category": "account"},
        {"input": "I was charged twice.", "expected_category": "billing"},
    ],
)

# From traces (columns are OpenInference span attributes)
spans_df = client.spans.export_to_df(
    space_id=space,
    project_name="my-llm-app",
    start_time=datetime.now() - timedelta(days=30),
    end_time=datetime.now(),
)
client.datasets.create(name="traces-v1", space=space, examples=spans_df)

Optional: include prompt template metadata in each row

Continuing from the Python example above, if each row needs to carry the prompt template and its filled variables, store them on the OpenInference prompt-template columns so Playground and code can map them consistently:

Python SDK v8

import json
import pandas as pd

PROMPT_TEMPLATE = """
You are an expert in the history of technological inventions.
Identify the individual or organization that created the following invention.

Invention: {invention}
"""

prompt_rows = pd.DataFrame(
    [
        {
            "attributes.llm.prompt_template.template": PROMPT_TEMPLATE,
            "attributes.llm.prompt_template.variables": json.dumps(
                {"invention": "Telephone"}
            ),
            "attributes.output.value": "Alexander Graham Bell",
        }
    ]
)

client.datasets.create(
    name="prompt-invention-dataset",
    space=space,
    examples=prompt_rows,
)

If you’re migrating from Python SDK v7 dataset APIs, see the datasets client migration guide for create_dataset() and update_dataset() replacements.

Manage your dataset

Add, edit, export, or delete rows as the app evolves. Datasets are versioned, and appends land in the latest version in place.

By Arize Skills

Use the arize-dataset skill to append, export, or inspect datasets without leaving your editor. Try asking your agent:

• “Append the rows in new_examples.csv to my support-regression dataset.”
• “Export the latest version of my support-tickets dataset so I can review it offline.”
• “Show me the schema and the first five rows of my support-qa-v1 dataset.”

Append new examples to an existing dataset from your coding agent with the arize-dataset skill

By Alyx

The Dataset Page Agent can append, annotate, or summarize datasets. Try:

• “Add the last 20 error spans to my regression dataset.”
• “Label the rows in this dataset with their expected category.”
• “Summarize how my regression dataset rows break down by failure type.”

Dataset Page Agent appending spans, annotating rows, or summarizing a dataset in Alyx.

By UI

Everything below happens on the dataset detail view, where you are working with a single dataset version at a time.

To add a row, click + Example and fill in the fields inline.

Add a row inline with + Example on the dataset detail view.

To edit, open any row in the table and change values in place. That is the natural place to add or correct expected outputs so regression rows become golden rows. To remove rows from the latest version, select them and click Delete.

When you want a file for offline review or to share outside AX, click Download as CSV on the dataset page.

Download a dataset as CSV from the dataset page

By Code

Use the Arize SDK to append or export dataset rows.

Python

import os
from arize import ArizeClient

client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])

# Append rows — lands in the latest version in place.
# Pass `dataset_version_id` to target a specific version.
client.datasets.append_examples(
    dataset="YOUR_DATASET_ID",
    examples=[{"input": "...", "expected_category": "billing"}],
)

# Export examples for offline analysis. `all=True` fetches every row.
examples_df = client.datasets.list_examples(
    dataset="YOUR_DATASET_ID",
    all=True,
).to_df()

For the full reference, including get, delete, DataFrame input, and pagination, see the Python datasets client and TypeScript datasets client.

Auto-add rows from evaluator labels

Once the dataset exists, set up rules that automatically add spans when they match your criteria. Auto-add rules keep the dataset current with what’s actually happening in production, without manual curation.

After you’ve set up an evaluator on a project, add a post-processing step that routes spans to a dataset based on the evaluator’s result. See Create evaluators for evaluator setup, then edit the evaluator configuration for your task.

Select the evaluator from the task configuration

Select Auto Add Spans to Dataset, then specify which eval labels should trigger the addition. For example, all spans where Correctness is Incorrect, or any span where the eval label is not null.

Configure auto-add rules from evaluator results

Run your first experiment

Measure whether a prompt, model, agent, or pipeline change actually improves your app.

Remote agent experiments

Run the dataset against a deployed agent over HTTP and collect every response.

Code experiments

Run the task in your own runtime, then log and compare results in Arize AX.

Prompt playground

Build a prompt, run it across the dataset, and compare variants without leaving the UI.

Further reading

• Improve your agent: a walkthrough that replays a failing production trace and validates the fix across a dataset.
• View and manage traces: find spans worth turning into regression cases.
• Human review: turn reviewer feedback into dataset rows.
• Labeling queues: collect labels at scale before you build or update a golden dataset.
• Complete experiments notebook: an end-to-end dataset, evaluator, and comparison walkthrough.