Why Public Benchmarks Lie: Building Your Own Eval Harness

Why public benchmarks lie — build your own eval harness from your task, and compare models fairly with a dataset, a task, and two evaluators.

When a new model tops a public leaderboard, it’s tempting to assume it’s the right choice for your application. But public benchmarks measure generic capabilities on generic data with a generic metric — and your task is narrow and specific. The only benchmark that predicts how a model performs on your task is a benchmark built from your task.

In this tutorial you’ll build that harness for an email text-extraction service and use it to compare two models fairly. You will:

• Build a small domain dataset of emails + their correct extractions — your benchmark, not a public one
• Define an extraction task with a fixed schema and prompt, parameterized only by the model
• Define two evaluators — string similarity and field-level accuracy — and see how they can rank the models differently
• Run the same harness across gpt-5.4-mini and the flagship gpt-5.5 and compare them fairly in Arize AX

---

Before you start

You need an Arize AX account and an OpenAI API key. Every code block below runs as a plain Python script.

pip install "arize>=8" arize-otel openai openinference-instrumentation-openai \
  jarowinkler pandas pydantic

Copy your Space ID and API key from your Arize AX Space Settings page and set them alongside your OpenAI key.

export ARIZE_SPACE_ID="<your-space-id>"
export ARIZE_API_KEY="<your-api-key>"
export OPENAI_API_KEY="<your-openai-api-key>"

Experiments in Arize AX

An Arize AX experiment is made of three elements: a dataset (the inputs and expected outputs), a task (run once per example), and one or more evaluators (score the task’s output). Hold the dataset and evaluators constant, swap only the model, and the comparison is fair by construction.

Set up the clients

One ArizeClient handles both datasets and experiments. Registering the tracer as well means the extraction calls the task makes are traced, so you can inspect an individual model call alongside its experiment score.

import os

from arize.client import ArizeClient
from arize.otel import register
from openai import OpenAI
from openinference.instrumentation.openai import OpenAIInstrumentor

MODEL_A = "gpt-5.4-mini"
MODEL_B = "gpt-5.5"

SPACE_ID = os.environ["ARIZE_SPACE_ID"]

# Tracing for the extraction calls themselves.
tracer_provider = register(
    space_id=SPACE_ID,
    api_key=os.environ["ARIZE_API_KEY"],
    project_name="email-extraction-eval-harness",
    set_global_tracer_provider=True,
)
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

# One client for datasets + experiments.
ax_client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])

# OpenAI client the extraction task calls.
client = OpenAI()

Build a domain dataset

This is the part public benchmarks can’t do for you. We hand-label a handful of emails the way our service actually sees them, each paired with the exact structured output we want back: sender, category, summary, action_required, and due_date.

EMAILS = [
    {
        "email": "From: Dana Whitfield <[e-mail]>\nSubject: Kickoff for Q3 redesign\n\nHi team, can we lock in the kickoff for the Q3 site redesign? I'm proposing Thursday July 10 at 2pm PT. Please confirm by end of week so I can send invites.",
        "expected": {
            "sender": "Dana Whitfield",
            "category": "meeting",
            "summary": "Proposes a Q3 redesign kickoff meeting on July 10 at 2pm PT.",
            "action_required": True,
            "due_date": "2025-07-10",
        },
    },
    {
        "email": "From: [e-mail]\nSubject: Invoice #88231 due\n\nYour CloudHost invoice #88231 for $4,200.00 is due on 2025-06-30. Please remit payment to avoid a service interruption.",
        "expected": {
            "sender": "[e-mail]",
            "category": "invoice",
            "summary": "CloudHost invoice #88231 for $4,200 is due June 30.",
            "action_required": True,
            "due_date": "2025-06-30",
        },
    },
    {
        "email": "From: Marcus Lee <[e-mail]>\nSubject: URGENT: dashboard down\n\nOur production dashboard has been returning 500 errors since this morning and our analysts are blocked. We need this resolved today. Ticket #4471 already open.",
        "expected": {
            "sender": "Marcus Lee",
            "category": "support_request",
            "summary": "Production dashboard is down with 500 errors; needs resolution today.",
            "action_required": True,
            "due_date": "none",
        },
    },
    {
        "email": "From: Priya Nair <[e-mail]>\nSubject: Re: enterprise pricing\n\nThanks for the deck. Leadership is interested but wants to see SOC 2 docs before we go further. No rush on timing.",
        "expected": {
            "sender": "Priya Nair",
            "category": "sales",
            "summary": "Interested in enterprise plan but needs SOC 2 docs before proceeding.",
            "action_required": True,
            "due_date": "none",
        },
    },
    {
        "email": "From: Tom Briggs <[e-mail]>\nSubject: Contract renewal by Aug 1\n\nOur MSA is up for renewal. To keep continuity, please sign and return the renewal addendum no later than August 1, 2025. Reach out with redlines.",
        "expected": {
            "sender": "Tom Briggs",
            "category": "sales",
            "summary": "MSA renewal addendum must be signed and returned by August 1.",
            "action_required": True,
            "due_date": "2025-08-01",
        },
    },
    {
        "email": "From: [e-mail]\nSubject: All systems operational\n\nThis is your weekly status digest. All monitored services were operational over the past 7 days with 100% uptime. No incidents reported.",
        "expected": {
            "sender": "[e-mail]",
            "category": "internal_update",
            "summary": "Weekly status digest: all services operational, 100% uptime, no incidents.",
            "action_required": False,
            "due_date": "none",
        },
    },
    {
        "email": "From: Sofia Reyes <[e-mail]>\nSubject: Need feedback on mockups\n\nDropped v2 of the mockups in the shared drive. Could you review and send comments before our sync on June 25? Want to finalize before the build phase.",
        "expected": {
            "sender": "Sofia Reyes",
            "category": "support_request",
            "summary": "Requests review of v2 mockups before the June 25 sync.",
            "action_required": True,
            "due_date": "2025-06-25",
        },
    },
]

Arize AX dataset examples are flat dicts, so each row is the email plus its five expected fields. The task and evaluators read the fields they need by name from each row.

from datetime import datetime, timezone

import pandas as pd

rows = [{"email": e["email"], **e["expected"]} for e in EMAILS]
df = pd.DataFrame(rows)
OUTPUT_KEYS = ["sender", "category", "summary", "action_required", "due_date"]

DATASET_NAME = f"email-extraction-{datetime.now(timezone.utc):%Y%m%d-%H%M%S}"
dataset = ax_client.datasets.create(name=DATASET_NAME, space=SPACE_ID, examples=df)
print(f"Uploaded {len(df)} examples to dataset '{DATASET_NAME}' (id: {dataset.id})")

The printed dataset name and id let you find it quickly in the Datasets tab in Arize AX.

Define the extraction task

The task is what we hold almost constant: the same schema, the same prompt, the same parsing — only the model changes. We use OpenAI’s structured outputs so every model returns the exact same shape. The experiment passes each example’s row as dataset_row; we read the email out of it.

from typing import Literal

from pydantic import BaseModel


class EmailExtraction(BaseModel):
    sender: str
    category: Literal["meeting", "invoice", "support_request", "sales", "internal_update"]
    summary: str
    action_required: bool
    due_date: str  # ISO date (YYYY-MM-DD) or the literal string "none"


PROMPT = (
    "Extract structured fields from the email below. "
    "category must be one of: meeting, invoice, support_request, sales, internal_update. "
    "due_date must be an ISO date (YYYY-MM-DD) if the email states a concrete deadline, "
    'otherwise the literal string "none". '
    "action_required is true if the email asks the recipient to do something.\n\nEMAIL:\n{email}"
)


def make_task(model: str):
    def task(dataset_row) -> dict:
        response = client.beta.chat.completions.parse(
            model=model,
            messages=[{"role": "user", "content": PROMPT.format(email=dataset_row["email"])}],
            response_format=EmailExtraction,
        )
        return response.choices[0].message.parsed.model_dump()

    return task

Sanity-check the task on a single example before spending a full experiment run on it.

print(make_task(MODEL_A)({"email": EMAILS[0]["email"]}))

Choose metrics that measure what you care about

This is where benchmarks quietly lie. We score the same outputs two ways: jaro_winkler (forgiving string similarity, the right tool for the free-text summary) and field_accuracy (exact match on the operational fields downstream code depends on). The two metrics measure different things, so they can rank the models differently — and when they disagree, the metric that should decide is the one tied to your downstream needs.

Each evaluator receives the task’s output and the example’s dataset_row, and returns an EvaluationResult (Arize AX requires a score plus a non-null label and explanation).

import json

import jarowinkler
from arize.experiments import EvaluationResult

OPERATIONAL_FIELDS = ["sender", "category", "action_required", "due_date"]


def _expected(dataset_row) -> dict:
    return {k: dataset_row[k] for k in OUTPUT_KEYS}


def jaro_winkler(output, dataset_row) -> EvaluationResult:
    score = jarowinkler.jarowinkler_similarity(
        json.dumps(output, sort_keys=True),
        json.dumps(_expected(dataset_row), sort_keys=True),
    )
    return EvaluationResult(
        score=score, label=f"{score:.2f}", explanation="JSON string similarity vs expected"
    )


def field_accuracy(output, dataset_row) -> EvaluationResult:
    expected = _expected(dataset_row)
    matches = sum(
        1
        for k in OPERATIONAL_FIELDS
        if str(output.get(k)).strip().lower() == str(expected[k]).strip().lower()
    )
    score = matches / len(OPERATIONAL_FIELDS)
    return EvaluationResult(
        score=score,
        label=f"{matches}/{len(OPERATIONAL_FIELDS)}",
        explanation=f"{matches} of {len(OPERATIONAL_FIELDS)} operational fields matched exactly",
    )


EVALUATORS = [jaro_winkler, field_accuracy]

The disagreement is easiest to see deterministically. Take two candidate extractions for the same invoice email: A has every operational field right but a fully reworded summary; B has an identical summary but its due_date is off by a day. field_accuracy prefers A (all operational fields correct), while jaro_winkler prefers B (it looks almost identical) — yet B’s one-day date slip is exactly what breaks downstream code. Same outputs, opposite rankings, and the strict metric is the right one to trust.

Run the same harness across models

Same dataset, same evaluators, same prompt — we change only the model argument. Each run(...) uploads its results to Arize AX and returns a results dataframe.

experiment_mini, results_mini = ax_client.experiments.run(
    name=f"{MODEL_A}-{DATASET_NAME}",
    dataset=DATASET_NAME,
    space=SPACE_ID,
    task=make_task(MODEL_A),
    evaluators=EVALUATORS,
)

experiment_full, results_full = ax_client.experiments.run(
    name=f"{MODEL_B}-{DATASET_NAME}",
    dataset=DATASET_NAME,
    space=SPACE_ID,
    task=make_task(MODEL_B),
    evaluators=EVALUATORS,
)

Compare fairly

Each run returns a results dataframe with an eval.<name>.score column per evaluator. Roll those up per metric, then open the dataset’s Experiments tab in Arize AX to compare the two runs example-by-example.

def average_scores(results) -> dict:
    return {
        "jaro_winkler": results["eval.jaro_winkler.score"].mean(),
        "field_accuracy": results["eval.field_accuracy.score"].mean(),
    }


summary = pd.DataFrame(
    {
        MODEL_A: average_scores(results_mini),
        MODEL_B: average_scores(results_full),
    }
)
print(summary)

View Results

Open the dataset’s Experiments tab to see the runs side by side. Each experiment is a row; each evaluator becomes its own score column (so field_accuracy sits next to jaro_winkler), alongside operational columns — average latency, cost, and error rate — that matter for a real model choice but never show up on a public leaderboard.

Sort by the metric tied to your downstream needs (field_accuracy) rather than the forgiving one, then click any row to drop into the example-level view: the input email, the model’s extraction, and each evaluator’s score for that single example. That’s where you see why one model wins — a due_date the model dropped, a sender it over-captured — instead of trusting an aggregate.

A public benchmark tells you how a model does on someone else’s task, with someone else’s metric. An Arize AX experiment built from your own data, with a metric tied to your downstream needs, gives you a number you can actually defend — and comparing models, prompts, or providers becomes just swapping one argument and reading it.