> ## Documentation Index > Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt > Use this file to discover all available pages before exploring further. # Why Public Benchmarks Lie: Building Your Own Eval Harness > Why public benchmarks lie — build your own eval harness from your task, and compare models fairly with a dataset, a task, and two evaluators. When a new model tops a public leaderboard, it's tempting to assume it's the right choice for *your* application. But public benchmarks measure **generic capabilities** on **generic data** with a **generic metric** — and your task is narrow and specific. The only benchmark that predicts how a model performs on your task is **a benchmark built from your task.** In this tutorial you'll build that harness for an email text-extraction service and use it to compare two models fairly. You will: * Build a small **domain dataset** of emails + their correct extractions — your benchmark, not a public one * Define an **extraction task** with a fixed schema and prompt, parameterized only by the model * Define **two evaluators** — string similarity *and* field-level accuracy — and see how they can rank the models differently * Run the **same harness** across `gpt-5.4-mini` and the flagship `gpt-5.5` and compare them fairly in Arize AX *** ## Before you start You need an [Arize AX account](https://app.arize.com/auth/join) and an OpenAI API key. Every code block below runs as a plain Python script. ```bash theme={null} pip install "arize>=8" arize-otel openai openinference-instrumentation-openai \ jarowinkler pandas pydantic ``` Copy your **Space ID** and **API key** from your Arize AX Space Settings page and set them alongside your OpenAI key. ```bash theme={null} export ARIZE_SPACE_ID="" export ARIZE_API_KEY="" export OPENAI_API_KEY="" ``` ## Experiments in Arize AX An Arize AX experiment is made of three elements: a **dataset** (the inputs and expected outputs), a **task** (run once per example), and one or more **evaluators** (score the task's output). Hold the dataset and evaluators constant, swap only the model, and the comparison is fair by construction. ## Set up the clients One `ArizeClient` handles both datasets and experiments. Registering the tracer as well means the extraction calls the task makes are traced, so you can inspect an individual model call alongside its experiment score. ```python theme={null} import os from arize.client import ArizeClient from arize.otel import register from openai import OpenAI from openinference.instrumentation.openai import OpenAIInstrumentor MODEL_A = "gpt-5.4-mini" MODEL_B = "gpt-5.5" SPACE_ID = os.environ["ARIZE_SPACE_ID"] # Tracing for the extraction calls themselves. tracer_provider = register( space_id=SPACE_ID, api_key=os.environ["ARIZE_API_KEY"], project_name="email-extraction-eval-harness", set_global_tracer_provider=True, ) OpenAIInstrumentor().instrument(tracer_provider=tracer_provider) # One client for datasets + experiments. ax_client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"]) # OpenAI client the extraction task calls. client = OpenAI() ``` ## Build a domain dataset This is the part public benchmarks can't do for you. We hand-label a handful of emails the way our service actually sees them, each paired with the exact structured output we want back: `sender`, `category`, `summary`, `action_required`, and `due_date`. ```python expandable theme={null} EMAILS = [ { "email": "From: Dana Whitfield \nSubject: Kickoff for Q3 redesign\n\nHi team, can we lock in the kickoff for the Q3 site redesign? I'm proposing Thursday July 10 at 2pm PT. Please confirm by end of week so I can send invites.", "expected": { "sender": "Dana Whitfield", "category": "meeting", "summary": "Proposes a Q3 redesign kickoff meeting on July 10 at 2pm PT.", "action_required": True, "due_date": "2025-07-10", }, }, { "email": "From: billing@cloudhost.com\nSubject: Invoice #88231 due\n\nYour CloudHost invoice #88231 for $4,200.00 is due on 2025-06-30. Please remit payment to avoid a service interruption.", "expected": { "sender": "billing@cloudhost.com", "category": "invoice", "summary": "CloudHost invoice #88231 for $4,200 is due June 30.", "action_required": True, "due_date": "2025-06-30", }, }, { "email": "From: Marcus Lee \nSubject: URGENT: dashboard down\n\nOur production dashboard has been returning 500 errors since this morning and our analysts are blocked. We need this resolved today. Ticket #4471 already open.", "expected": { "sender": "Marcus Lee", "category": "support_request", "summary": "Production dashboard is down with 500 errors; needs resolution today.", "action_required": True, "due_date": "none", }, }, { "email": "From: Priya Nair \nSubject: Re: enterprise pricing\n\nThanks for the deck. Leadership is interested but wants to see SOC 2 docs before we go further. No rush on timing.", "expected": { "sender": "Priya Nair", "category": "sales", "summary": "Interested in enterprise plan but needs SOC 2 docs before proceeding.", "action_required": True, "due_date": "none", }, }, { "email": "From: Tom Briggs \nSubject: Contract renewal by Aug 1\n\nOur MSA is up for renewal. To keep continuity, please sign and return the renewal addendum no later than August 1, 2025. Reach out with redlines.", "expected": { "sender": "Tom Briggs", "category": "sales", "summary": "MSA renewal addendum must be signed and returned by August 1.", "action_required": True, "due_date": "2025-08-01", }, }, { "email": "From: alerts@statuspage.io\nSubject: All systems operational\n\nThis is your weekly status digest. All monitored services were operational over the past 7 days with 100% uptime. No incidents reported.", "expected": { "sender": "alerts@statuspage.io", "category": "internal_update", "summary": "Weekly status digest: all services operational, 100% uptime, no incidents.", "action_required": False, "due_date": "none", }, }, { "email": "From: Sofia Reyes \nSubject: Need feedback on mockups\n\nDropped v2 of the mockups in the shared drive. Could you review and send comments before our sync on June 25? Want to finalize before the build phase.", "expected": { "sender": "Sofia Reyes", "category": "support_request", "summary": "Requests review of v2 mockups before the June 25 sync.", "action_required": True, "due_date": "2025-06-25", }, }, ] ``` Arize AX dataset examples are flat dicts, so each row is the email plus its five expected fields. The task and evaluators read the fields they need by name from each row. ```python theme={null} from datetime import datetime, timezone import pandas as pd rows = [{"email": e["email"], **e["expected"]} for e in EMAILS] df = pd.DataFrame(rows) OUTPUT_KEYS = ["sender", "category", "summary", "action_required", "due_date"] DATASET_NAME = f"email-extraction-{datetime.now(timezone.utc):%Y%m%d-%H%M%S}" dataset = ax_client.datasets.create(name=DATASET_NAME, space=SPACE_ID, examples=df) print(f"Uploaded {len(df)} examples to dataset '{DATASET_NAME}' (id: {dataset.id})") ``` The printed dataset name and id let you find it quickly in the **Datasets** tab in Arize AX. ## Define the extraction task The task is what we hold *almost* constant: the same schema, the same prompt, the same parsing — **only the model changes**. We use OpenAI's structured outputs so every model returns the exact same shape. The experiment passes each example's row as `dataset_row`; we read the email out of it. ```python expandable theme={null} from typing import Literal from pydantic import BaseModel class EmailExtraction(BaseModel): sender: str category: Literal["meeting", "invoice", "support_request", "sales", "internal_update"] summary: str action_required: bool due_date: str # ISO date (YYYY-MM-DD) or the literal string "none" PROMPT = ( "Extract structured fields from the email below. " "category must be one of: meeting, invoice, support_request, sales, internal_update. " "due_date must be an ISO date (YYYY-MM-DD) if the email states a concrete deadline, " 'otherwise the literal string "none". ' "action_required is true if the email asks the recipient to do something.\n\nEMAIL:\n{email}" ) def make_task(model: str): def task(dataset_row) -> dict: response = client.beta.chat.completions.parse( model=model, messages=[{"role": "user", "content": PROMPT.format(email=dataset_row["email"])}], response_format=EmailExtraction, ) return response.choices[0].message.parsed.model_dump() return task ``` Sanity-check the task on a single example before spending a full experiment run on it. ```python theme={null} print(make_task(MODEL_A)({"email": EMAILS[0]["email"]})) ``` ## Choose metrics that measure what you care about This is where benchmarks quietly lie. We score the **same** outputs two ways: `jaro_winkler` (forgiving string similarity, the right tool for the free-text `summary`) and `field_accuracy` (exact match on the *operational* fields downstream code depends on). The two metrics measure different things, so they *can* rank the models differently — and when they disagree, the metric that should decide is the one tied to your downstream needs. Each evaluator receives the task's `output` and the example's `dataset_row`, and returns an `EvaluationResult` (Arize AX requires a score plus a non-null label and explanation). ```python expandable theme={null} import json import jarowinkler from arize.experiments import EvaluationResult OPERATIONAL_FIELDS = ["sender", "category", "action_required", "due_date"] def _expected(dataset_row) -> dict: return {k: dataset_row[k] for k in OUTPUT_KEYS} def jaro_winkler(output, dataset_row) -> EvaluationResult: score = jarowinkler.jarowinkler_similarity( json.dumps(output, sort_keys=True), json.dumps(_expected(dataset_row), sort_keys=True), ) return EvaluationResult( score=score, label=f"{score:.2f}", explanation="JSON string similarity vs expected" ) def field_accuracy(output, dataset_row) -> EvaluationResult: expected = _expected(dataset_row) matches = sum( 1 for k in OPERATIONAL_FIELDS if str(output.get(k)).strip().lower() == str(expected[k]).strip().lower() ) score = matches / len(OPERATIONAL_FIELDS) return EvaluationResult( score=score, label=f"{matches}/{len(OPERATIONAL_FIELDS)}", explanation=f"{matches} of {len(OPERATIONAL_FIELDS)} operational fields matched exactly", ) EVALUATORS = [jaro_winkler, field_accuracy] ``` The disagreement is easiest to see deterministically. Take two candidate extractions for the same invoice email: **A** has every operational field right but a fully reworded `summary`; **B** has an identical summary but its `due_date` is off by a day. `field_accuracy` prefers A (all operational fields correct), while `jaro_winkler` prefers B (it looks almost identical) — yet B's one-day date slip is exactly what breaks downstream code. **Same outputs, opposite rankings, and the strict metric is the right one to trust.** ## Run the same harness across models Same dataset, same evaluators, same prompt — we change only the `model` argument. Each `run(...)` uploads its results to Arize AX and returns a results dataframe. ```python theme={null} experiment_mini, results_mini = ax_client.experiments.run( name=f"{MODEL_A}-{DATASET_NAME}", dataset=DATASET_NAME, space=SPACE_ID, task=make_task(MODEL_A), evaluators=EVALUATORS, ) experiment_full, results_full = ax_client.experiments.run( name=f"{MODEL_B}-{DATASET_NAME}", dataset=DATASET_NAME, space=SPACE_ID, task=make_task(MODEL_B), evaluators=EVALUATORS, ) ``` ## Compare fairly Each run returns a results dataframe with an `eval..score` column per evaluator. Roll those up per metric, then open the dataset's **Experiments** tab in Arize AX to compare the two runs example-by-example. ```python theme={null} def average_scores(results) -> dict: return { "jaro_winkler": results["eval.jaro_winkler.score"].mean(), "field_accuracy": results["eval.field_accuracy.score"].mean(), } summary = pd.DataFrame( { MODEL_A: average_scores(results_mini), MODEL_B: average_scores(results_full), } ) print(summary) ``` ## View Results Open the dataset's **Experiments** tab to see the runs side by side. Each experiment is a row; each evaluator becomes its own **score column** (so `field_accuracy` sits next to `jaro_winkler`), alongside operational columns — average **latency**, **cost**, and **error rate** — that matter for a real model choice but never show up on a public leaderboard. Sort by the metric tied to your downstream needs (`field_accuracy`) rather than the forgiving one, then click any row to drop into the example-level view: the input email, the model's extraction, and each evaluator's score for that single example. That's where you *see why* one model wins — a `due_date` the model dropped, a sender it over-captured — instead of trusting an aggregate. A public benchmark tells you how a model does on *someone else's* task, with *someone else's* metric. An Arize AX experiment built from your own data, with a metric tied to your downstream needs, gives you a number you can actually defend — and comparing models, prompts, or providers becomes just swapping one argument and reading it.