October 2026

Find and fix eval gaps with the Eval Health agent

October 21, 2026

New  Evaluators

Eval Health is a managed agent that reviews the evals on a tracing project and proposes changes for you to accept. It reads the project’s recent spans, tasks, and evaluators, then files an issue for each coverage gap or broken eval it finds, along with a proposed action:

• Create: add an eval for a part of your app that isn’t covered yet.
• Fix: repair an eval in place, such as its column mappings, query filter, or template and labels.
• Delete: remove an eval that’s no longer producing useful signal.
• Optimize: cut eval cost by converting an LLM judge to a code eval or suggesting a different judge model.

Nothing changes until you accept it. Every proposal is ready to run the moment you approve it, and a run that finds sufficient coverage proposes nothing. Find it by adding the Eval Health Agent on any tracing project.

Score whole conversations with trace and session remote evaluators

October 21, 2026

Improvement  Evaluators

At trace and session scope, a remote evaluator now receives each mapped variable as an array with one value per span instead of a single comma-joined string, so your endpoint can score an entire conversation in one call. Spans are ordered by start time, each trace’s spans stay together, and every variable’s array covers the same spans in the same order, so index i refers to the same span across all variables. This is the default for every trace and session remote evaluator; span scope is unchanged.

• Spans with no value appear as null, so each variable’s array stays aligned with the others.
• Variables that exceed the size cap are truncated, and metadata.truncated_variables lists which ones were cut.

The code example below shows the input payload a trace-scoped remote evaluator now receives, with one array entry per span in the trace:

{
  "input": {
    "user_input": ["Can you tell me my deductible?", null, null],
    "agent_response": ["Let me check, one moment.", null, "Your individual deductible is $1,500, and $780 is met."]
  }
}

Learn more about remote evaluators.

Correlate evaluation scores with their task and run

October 21, 2026

New  SDKs and REST APIs

Listing spans and traces now returns each evaluation’s metadata, including the task_id and run_id of the evaluation task that produced the score, so you can tie a score back to the task and run that generated it without a separate Flight export. The field is returned on both /v2/spans and /v2/traces, and it is omitted when an evaluation has no metadata.

The code example below lists spans and reads the task and run behind each evaluation:

import { listSpans } from "@arizeai/ax-client";

const { data: spans } = await listSpans({
  project: "My Project",
  space: "my-space",
  limit: 100,
});

for (const span of spans) {
  for (const evaluation of span.evaluations) {
    // metadata is populated for scores written by an evaluation task
    if (evaluation.metadata) {
      console.log(evaluation.name, evaluation.metadata.task_id, evaluation.metadata.run_id);
    }
  }
}

Learn more about listing spans.

Run trace and session evaluators on datasets

October 20, 2026

New  Datasets and Experiments

Trace-level and session-level evaluators can now run over a dataset, not only over live projects and experiments, so you can score multi-span traces and full sessions from data you have already collected. The evaluator groups the dataset’s rows into trace or session units and writes one verdict per unit instead of scoring each row on its own. If you created such a task through the API or SDK, it switches from per-row to per-unit scoring.

Learn more about trace and session evaluations.

Connect Perplexity and Cloudflare as AI providers

October 20, 2026

New  Models and Integrations

Perplexity and Cloudflare are now available as AI providers in Arize AX, so you can use them for prompts and evaluations alongside your existing providers. Cloudflare is also available as a Decision Judge provider, which means you can run Decision Judge evaluations on Cloudflare models.

Learn more about AI provider integrations.

Track Claude Code spend by model and user

October 20, 2026

New  Models and Integrations

A new Anthropic Console cost connector pulls Claude Code usage and spend from Anthropic’s Claude Code Analytics API, so you can track what your organization spends on Claude Code and break it down by model and by user. Connect it with an Anthropic Admin key to import daily token usage and estimated cost.

Learn more about tracking costs.

Restrict Alyx to one AI provider and model

October 20, 2026

New  Alyx

Space admins can now lock Alyx to a single AI provider and model from Space Settings, so everyone in the space works through an approved model and Alyx rejects requests for any other one. Once a model is set, members see it fixed in the Alyx model picker.

Learn more about Alyx.

Edit a saved dashboard’s scope and name

October 20, 2026

Improvement  Dashboards and Visualizations

You can now rename an account dashboard and change its saved scope after creating it, using the Edit button on the dashboard. Point it at the entire account, specific organizations, or specific spaces without rebuilding the dashboard, and its default time range is preserved.

Learn more about dashboards.

Fixes and improvements

October 20–21, 2026

Evaluators

• Improvement See how many rows an evaluation skipped and why: run summaries now count rows skipped for null input values and name the columns responsible, so successes, skips, and failures account for every row fetched.

Datasets and Experiments

• Fix Stop a task run and get an accurate status: the run is marked cancelled, keeps the counts it scored before stopping, and a fully-skipped run reports the reason.

Playground

• Fix Saving a prompt version to the hub now persists the full set of model parameters, including unchanged defaults, so a reloaded prompt reproduces the original configuration.
• Fix Open prompts saved from spans that contain tool calls in the prompt playground.

Tracing and Sessions

• Fix Build trace filters that hold: NULL checks commit as is empty or exists, OR connectors stay OR after you clear a row, pasted SQL filters parse correctly, and View Traces from the agent graph keeps your filter.

Annotations

• Fix View a session’s traces and compare them from an annotation queue: View All Traces uses the session’s own time window, and trace comparison works on queue routes.

Custom Metrics

• Fix Trigger a managed agent on any custom metric in its space, not only a metric defined in the trigger’s own project.

Agents

• Improvement Disconnect a connected Slack workspace from your account on the Slack Agents settings page, rather than only removing individual channel bindings.

Alyx

• Fix Accept an Alyx custom metric proposal on production-only and multi-environment models: Alyx now validates a proposed metric against the environment the target model actually uses.
• Improvement Attach up to 16 custom request-metadata pairs to Alyx calls on an AWS Bedrock integration, so your own Bedrock logs attribute usage to Arize AX users.

Models and Integrations

• Improvement Track OpenAI spend using the per-API-key costs reported by OpenAI’s Costs API, so per-key and per-project totals match your actual bill.

Build and run Decision Judge evaluators with Alyx

October 19, 2026

New  Alyx

You can now ask Alyx to create, edit, and run Decision Judge evaluators from chat, so you can go from a description of what to check to scored spans. Alyx fills in the state template and typed questions in the Decision Judge editor, and nothing saves until you accept it. It works with any Decision Judge provider in your space, including Jev-as-a-Judge on TypeSafe AI; name one, such as OpenAI, to use it.

• Run: Alyx reuses a matching evaluator or proposes a new one, creates a project task, checks its column mappings, and runs it over historical LLM spans.
• Edit: Accepted changes to the state template, questions, or description save as a new version.
• Explain: Alyx shows an evaluator’s model, integration, state template, and the criteria for each question.

Learn more about Decision Judge.

Run Decision Judge evals on OpenAI decision models

October 16, 2026

Improvement  Evaluators

Jev-as-a-Judge is now Decision Judge, a provider-neutral evaluator type. The same shared state and typed boolean, choice, and score questions can run on TypeSafe AI’s jev-latest or OpenAI’s gpt-6-luna through your existing OpenAI integration. Pick the provider from the AI Provider dropdown in the evaluator editor; only integrations in your space that can run Decision Judge evaluations are listed. Most Arize eval templates now have a Decision Judge version, loaded by selecting a TypeSafe AI model on the template card.

Existing Jev-as-a-Judge evaluators keep working unchanged. Links to the old docs page redirect to the new one.

Learn more about Decision Judge.

Reference a range monitor’s second threshold in alert notes

October 13, 2026

Improvement  Webhooks and Events

When you compose a monitor’s alert note, you can now insert the second threshold of a range monitor with the {{threshold2.operator}} and {{threshold2.value}} variables, so an alert can state both bounds of the range that triggered it. For example, a note of Metric is {{threshold2.operator}} {{threshold2.value}} renders the evaluated second bound in each alert. The two variables appear in the variable picker on the Monitor Notes field, in the Notifications section of the monitor configuration, and only when the monitor’s threshold mode is set to range. The existing {{threshold.operator}} and {{threshold.value}} variables continue to reference the first threshold.

For more information, refer to the monitor notifications documentation.

Filter traces by clicking a bar on an eval chart

October 12, 2026

Improvement  Tracing and Sessions

On the Evals and Metrics tab in LLM tracing, you can now click a bar on an eval chart card to filter the traces by that eval label. The full column is clickable, so you can also filter by short bars and labels such as N/A that were previously hard to target. The same action is available as a Filter by label submenu in each card’s overflow menu.

Learn more about viewing and managing traces.

Turn a Signal issue into a running evaluator from the issue

October 9, 2026

Improvement  Alyx

Click Create Evaluator on a Signal issue, and Alyx builds an evaluator for that failure pattern and starts scoring new traces with it. Alyx creates a continuous task with no historical backfill and sets up its filters and column mappings automatically. If the chat is closed, Alyx works in the background; when it finishes, View Task in the notification opens the task on the current page.

Learn more about Signal.

Fixes and improvements

October 8–14, 2026

• Fix  Tracing and Sessions Keep building span filters without losing your in-progress text: Add Filter in the trace filter bar now appends the next filter instead of replacing the one you are editing.
• Fix  Tracing and Sessions Display and filter tracing columns whose names contain dots, such as a workflow.name metadata key, a dotted prompt-template variable, or an annotation like quality.v2.
• Fix  SDKs and REST APIs Call the Python SDK against a server of a different version without errors when a REST response omits fields the client expects.
• Fix  CLI Commands Run ax ai-integrations list to list a space’s AI integrations even when the space contains a TypeSafe AI integration.

Create and manage webhooks from the REST API, SDKs, and CLI

October 7, 2026

New  Webhooks and Events

You can now create webhooks and subscribe them to prompt and evaluator events from the REST API, the Python, TypeScript, and Go SDKs, or the ax CLI, so you can script event delivery to your own systems, such as a CI job that runs when a prompt version gets a new label. A webhook is an HTTPS endpoint in your organization that authenticates each delivery with a bearer token or an HMAC-SHA256 signature. A subscription connects one webhook to one event on one prompt or evaluator. These endpoints are in alpha and may change without notice.

• Events: PROMPT_VERSION_CREATED, PROMPT_VERSION_LABELED, and PROMPT_VERSION_UNLABELED for prompts; EVALUATOR_VERSION_CREATED for evaluators.
• Tests and delivery history: Send a test delivery to a bearer-token webhook, and list each webhook’s delivery attempts, including retries.
• Signing secrets: Arize AX returns an HMAC-SHA256 webhook’s signing secret only once, in the create response, so store it when you create the webhook.
• Versions: Python arize 8.54.0 or later (client.webhooks), TypeScript @arizeai/ax-client 1.30.0 or later, Go SDK v2 v0.24.0 or later, and arize-ax-cli 0.39.0 or later (ax webhooks).

The code example below creates a bearer-token webhook and subscribes it to new versions of a prompt:

Python

import os

from arize import ArizeClient
from arize.webhooks.types import (
    WebhookAuthType,
    WebhookEventType,
    WebhookSourceType,
)

client = ArizeClient()  # reads ARIZE_API_KEY from the environment

webhook = client.webhooks.create(
    organization="your-org-name-or-id",
    name="prompt-release-hook",
    url="https://hooks.example.com/arize",
    auth_type=WebhookAuthType.BEARER,
    # Sent as the full Authorization header value on each delivery
    auth_token=f"Bearer {os.environ['WEBHOOK_AUTH_TOKEN']}",
)

subscription = client.webhooks.create_subscription(
    webhook=webhook.id,
    source_type=WebhookSourceType.PROMPT,
    source_id="your-prompt-id",
    event=WebhookEventType.PROMPT_VERSION_CREATED,
)

print(subscription.id)

TypeScript

import { createWebhook, createWebhookSubscription } from "@arizeai/ax-client";

// The client reads ARIZE_API_KEY from the environment
const webhook = await createWebhook({
  organization: "your-org-name-or-id",
  name: "prompt-release-hook",
  url: "https://hooks.example.com/arize",
  authType: "BEARER",
  // Sent as the full Authorization header value on each delivery
  authToken: `Bearer ${process.env.WEBHOOK_AUTH_TOKEN}`,
});

const subscription = await createWebhookSubscription({
  webhook: webhook.id,
  sourceType: "PROMPT",
  sourceId: "your-prompt-id",
  event: "PROMPT_VERSION_CREATED",
});

console.log(subscription.id);

Go

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/Arize-ai/client-go-v2/arize"
    "github.com/Arize-ai/client-go-v2/arize/webhooks"
)

func main() {
    // An empty Config reads ARIZE_API_KEY from the environment
    client, err := arize.NewClient(arize.Config{})
    if err != nil {
        log.Fatal(err)
    }
    ctx := context.Background()

    wh, err := client.Webhooks.Create(ctx, webhooks.CreateRequest{
        Organization: "your-org-name-or-id",
        Name:         "prompt-release-hook",
        URL:          "https://hooks.example.com/arize",
        AuthType:     webhooks.WebhookAuthTypeBearer,
        // Sent as the full Authorization header value on each delivery
        AuthToken: "Bearer " + os.Getenv("WEBHOOK_AUTH_TOKEN"),
    })
    if err != nil {
        log.Fatal(err)
    }

    sub, err := client.Webhooks.CreateSubscription(ctx, webhooks.CreateSubscriptionRequest{
        Webhook:    wh.Id,
        SourceType: webhooks.WebhookSourceTypePrompt,
        SourceID:   "your-prompt-id",
        Event:      webhooks.WebhookEventTypePromptVersionCreated,
    })
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(sub.Id)
}

CLI

# BEARER is the default auth type
ax webhooks create \
  --organization "your-org-name-or-id" \
  --name "prompt-release-hook" \
  --url https://hooks.example.com/arize \
  --auth-token "Bearer $WEBHOOK_AUTH_TOKEN"

# A webhook name needs --organization to resolve
ax webhooks subscriptions create \
  --webhook "prompt-release-hook" \
  --organization "your-org-name-or-id" \
  --source-type PROMPT \
  --source-id "your-prompt-id" \
  --event PROMPT_VERSION_CREATED

Learn more about webhooks, the ax CLI, and the Python, TypeScript, and Go SDK references.

Filter spans by nested JSON attributes with path typeahead

October 6, 2026

New  Tracing and Sessions

When you build a span filter on the tracing page, Arize AX now suggests nested JSON attribute paths as you type, so you can drill into structured span attributes without recalling the exact path. Clicking a nested attribute value in a span adds it to your filter directly. The feature is available on every account tier across cloud and self-hosted deployments.

For more information, refer to the view and manage traces documentation.

Export full datasets and experiments to Parquet from the Python SDK

October 5, 2026

New  Datasets and Experiments

The Python SDK can now stream an entire dataset or experiment to a Parquet file on disk, batch by batch, so exports larger than available memory complete without loading every row into memory at once. The file holds the same rows and columns as list_examples(all=True) and list_runs(all=True), and is written to a temporary .partial file and renamed on completion, so a failed export leaves any existing file untouched.

The code example below streams a full dataset export and a full experiment export to Parquet files:

from arize import ArizeClient

client = ArizeClient(api_key="YOUR_API_KEY")

# Stream every example in a dataset to a Parquet file on disk.
client.datasets.export_to_parquet(
    dataset="my-dataset",       # dataset ID or name
    space="my-space",           # required when dataset is passed by name
    path="examples.parquet",
)

# Stream every run in an experiment to a Parquet file on disk.
client.experiments.export_to_parquet(
    experiment="my-experiment", # experiment ID or name
    path="runs.parquet",
)

Learn more about exporting datasets and experiments.

Select GPT-6.1 Sol and Claude Sonnet 5.5 in the Playground

October 1, 2026

New  Models and Integrations

Two new models are available for prompts and evaluations in Arize AX: OpenAI’s GPT-6.1 Sol and Anthropic’s Claude Sonnet 5.5. Select either from the model picker in the Playground to run prompts, build evaluators, or power online tasks.

• GPT-6.1 Sol: available on OpenAI and Azure OpenAI, with a 128,000-token completion limit and configurable reasoning effort.
• Claude Sonnet 5.5: available on Anthropic, AWS Bedrock, and Google Vertex AI, with a 128,000-token completion limit and adaptive reasoning effort.

Learn more about supported model providers.

Fixes and improvements

September 29 – October 6, 2026

Tracing and Sessions

• Fix View harness projects, such as claude-code, on the Tracing Projects page and through the GET /v2/projects REST endpoint again; each is marked with a Harness label so it is not mistaken for a production project.
• Fix See a consistent score on task-less evaluator cards in the Tracing project Metrics tab, whether the card is pinned or unpinned.
• Fix Read accurate session metrics, such as total tokens, for sessions whose traces contain more than one root span.

Evaluators

• Fix See accurate evaluator scores on experiments, where nested metadata columns are no longer counted toward an evaluator’s score.
• Improvement Filter evaluation tasks by whether they run continuously, using the new run-mode dropdown on the Tasks page and the Running Eval Tasks tab of the Evaluators page.

Alyx

• Fix See the true dataset row count when Alyx previews a dataset, so it recognizes when a preview covers only part of the data and filters server-side.

Dashboards and Visualizations

• Fix Export dashboard widget data and view pivot tables with your active filters applied, so the exported and displayed rows match what you filtered to.

Annotations

• Fix Open annotation queue and dataset records that include an annotation named notes, which previously failed to load.

Webhooks and Events

• Fix Convert webhook list responses to a DataFrame with to_df(), which now returns one row per webhook, delivery attempt, or subscription.

Models and Integrations

• Fix Run Claude models selected through AWS Bedrock inference-profile ARNs or custom profile names, including regional profile IDs (eu., apac., jp., au., and us-gov.), in the Playground and LLM-as-a-judge evaluators, with the required token limits sent on every request.