Evaluating RAG Retrieval Quality and Correctness

Trace, evaluate, diagnose, and improve a retrieval-augmented generation (RAG) application in Arize AX, from retrieval relevance to chunking experiments.

Google Colab

Follow an end-to-end example of tracing and evaluating RAG

Overview

In this tutorial, you’ll build a retrieval-augmented generation (RAG) application with LlamaIndex, trace it in Arize AX, then evaluate, diagnose, and improve its retrieval quality. The application is a simple question-answering chatbot that answers from a single sample document.

This tutorial is intended for AI engineers building RAG systems. It assumes you have basic knowledge of:

• Python and running notebooks
• How RAG works at a high level (retrieval + generation)
• LLM application concepts (embeddings, vector stores, prompts)

By the end of this tutorial, you’ll be able to:

• Instrument a RAG application and send traces to Arize AX
• Evaluate retrieval relevance and answer correctness with LLM-as-a-judge
• Diagnose the three common retrieval failure modes using embeddings
• Run experiments to improve retrieval across chunk size, overlap, and k

Background: how RAG retrieval works and where it fails

RAG connects your own data to an LLM. A user asks a question, an embedding is generated from the query, the most relevant context in your knowledge base is retrieved, and that context is added to the prompt sent to the LLM.

When a RAG application returns a bad answer, it usually traces back to one of three retrieval failure modes:

1. Bad response. The end answer is wrong or unhelpful, often surfaced by negative user feedback or low eval scores. This is usually a symptom of one of the failures below.
2. Missing context. The retriever couldn’t find any documents close enough to the query, meaning users are asking about topics missing from your corpus.
3. Most similar is not most relevant. A document had the closest embedding to the query but wasn’t actually the most relevant one to answer it.

For a walkthrough of these concepts, watch the video:

Before you start

Before you begin, you should have:

• An Arize AX account. Sign up here and get your space ID and API key
• An OpenAI API key
• An LLM provider configured in Arize AX, required to run no-code evaluations in Step 2
• The required packages installed:

pip install "arize>=8.0.0" arize-otel openinference-instrumentation-llama-index llama-index arize-phoenix-evals

Step 1: Trace your RAG app

Arize AX auto-instruments many RAG stacks, including LlamaIndex and LangChain, and supports manual instrumentation for custom ones. This tutorial uses LlamaIndex.

First, set your API keys as environment variables. Every step reads credentials from here, so run this once at the start:

import os
from getpass import getpass

os.environ["ARIZE_SPACE_ID"] = getpass("Enter your Arize space ID")
os.environ["ARIZE_API_KEY"] = getpass("Enter your Arize API key")
os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key")

Set up tracing with arize-otel, a convenience package that configures OpenTelemetry alongside OpenInference auto-instrumentation, which maps LLM metadata to a standardized set of trace and span attributes. This sends every LlamaIndex call to Arize AX:

from arize.otel import register
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor

tracer_provider = register(
    space_id=os.environ["ARIZE_SPACE_ID"],
    api_key=os.environ["ARIZE_API_KEY"],
    project_name="rag-cookbook",
)
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)

Next, give the chatbot a knowledge base to answer from. We’ll use an essay by Paul Graham, the sample document from LlamaIndex’s starter tutorial. Download it into a local data/ folder:

!mkdir data
!wget "https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt" -O data/paul_graham_essay.txt

Load the document, build a vector index over it, and expose that index as a query engine. This query engine is the RAG application you’ll trace and evaluate:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI

# Load the sample document
documents = SimpleDirectoryReader("data").load_data()

# Create Vector Store Index
index = VectorStoreIndex.from_documents(documents)

# Turn it into a query engine using gpt-4.1-mini
query_engine = index.as_query_engine(llm=OpenAI(model="gpt-4.1-mini"))

# Create a query
response = query_engine.query("What did Paul Graham work on?")
print(response)

This creates a trace in Arize AX, which visualizes inputs, outputs, retrieved chunks, embeddings, and LLM prompt parameters.

Send a few more queries so you have several traces to evaluate in the next step:

for question in [
    "What did Paul Graham do growing up?",
    "What did Paul Graham study in college?",
    "What did Paul Graham do at Y Combinator?",
    "Why did Paul Graham start painting?",
]:
    query_engine.query(question)

Step 2: Evaluate retrieval quality and correctness

Now that traces are flowing into Arize AX, evaluate them so you don’t have to manually inspect every trace. We’ll score two things:

• Retrieval relevance: is the retrieved context relevant to the question? This is judged on the retriever span, which carries both the query (input) and the retrieved documents.
• Answer correctness: is the final answer correct given the retrieved context? This draws on two spans: the query and context come from the retriever span, and the answer comes from the LLM span.

Because the inputs to these evaluators live on different spans, retrieval relevance is a span-level eval on retriever spans, while correctness is a trace-level eval that reads across the retriever and LLM spans in a trace.

Arize AX uses the Phoenix Evals library for LLM-as-a-judge, and its prebuilt templates (including RAG relevance, QA correctness, and hallucination) are available both in the UI and in code.

Option A: Run a no-code evaluation task (recommended)

Create a task that runs an evaluator from the Eval Hub directly on your traces, with no export and no dataframe. You can do this from the UI, from Alyx, or the ax CLI. In the UI, click New Task from the Evaluators page, select your rag-cookbook project as the data source, then Add Evaluator and pick a prebuilt template from the Eval Hub.

Retrieval relevance (span-level). Add the prebuilt Retrieval Relevance template, scope it to span level, and filter to retriever spans with attributes.openinference.span.kind = 'RETRIEVER'. Both variables come from that one span:

Variable	Column
{input}	input
{reference}	attributes.retrieval.documents

Answer correctness (trace-level). Correctness needs the question, the retrieved context, and the answer, which live on different spans. Add the prebuilt QA Correctness template, scope it to trace level, and use a multi-span query so each variable reads from the right span role:

Subquery	Filter
A	attributes.openinference.span.kind = 'RETRIEVER'
B	attributes.openinference.span.kind = 'LLM' AND status_code = 'OK'

Expression A => B (the retriever is the direct parent of the LLM span), with this variable mapping:

Variable	Column	Subquery
{question}	input	A
{context}	attributes.retrieval.documents	A
{answer}	output	B

For each task, set the cadence (one-time backfill or run continuously), sampling rate, and any filters, then click Create Task. Results attach automatically to your spans in the Tracing view. For the full task workflow, sampling guidance, and multi-span queries, see Run online evals on traces.

Option B: Run evaluations in code

Use this when you need to run evals on large datasets, incorporate external data, or want full control over execution and cost. Export your spans, run evals with Phoenix Evals, and log the results back onto your spans.

First, export your spans and split them by role:

import os
from datetime import datetime, timedelta
from arize import ArizeClient

client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])

end_time = datetime.now()
start_time = end_time - timedelta(days=1)

spans_df = client.spans.export_to_df(
    space_id=os.environ["ARIZE_SPACE_ID"],
    project_name="rag-cookbook",
    start_time=start_time,
    end_time=end_time,
)

# Retriever spans carry the query and the retrieved documents
retriever_df = spans_df[
    spans_df["attributes.openinference.span.kind"] == "RETRIEVER"
].copy()

Define the judge and a retrieval-relevance evaluator, then run it. The template placeholders ({input}, {reference}) are filled from same-named columns in each row, so map the span attributes onto those column names first:

from phoenix.evals import LLM, create_classifier, evaluate_dataframe

retriever_df["input"] = retriever_df["attributes.input.value"]
retriever_df["reference"] = retriever_df["attributes.retrieval.documents"]

llm = LLM(provider="openai", model="gpt-4.1")

RELEVANCE_TEMPLATE = """You are comparing a reference text to a question and trying to determine
if the reference text contains information relevant to answering the question. Here is the data:
    [BEGIN DATA]
    ************
    [Question]: {input}
    ************
    [Reference text]: {reference}
    [END DATA]
Is the Reference text relevant to answering the Question? Answer "relevant" or "unrelated"."""

relevance_evaluator = create_classifier(
    name="retrieval_relevance",
    llm=llm,
    prompt_template=RELEVANCE_TEMPLATE,
    choices={"relevant": 1.0, "unrelated": 0.0},
)

results_df = evaluate_dataframe(
    dataframe=retriever_df,
    evaluators=[relevance_evaluator],
)

Then log the results back onto your spans. to_annotation_dataframe explodes the scores into label, score, and explanation columns and carries the context.span_id through; update_evaluations expects those under the eval.<name>.* convention:

from phoenix.evals.utils import to_annotation_dataframe

annotation_df = to_annotation_dataframe(results_df, ["retrieval_relevance"])

eval_df = annotation_df.rename(columns={
    "label": "eval.retrieval_relevance.label",
    "score": "eval.retrieval_relevance.score",
    "explanation": "eval.retrieval_relevance.explanation",
})[["context.span_id",
    "eval.retrieval_relevance.label",
    "eval.retrieval_relevance.score",
    "eval.retrieval_relevance.explanation"]]

client.spans.update_evaluations(
    space_id=os.environ["ARIZE_SPACE_ID"],
    project_name="rag-cookbook",
    dataframe=eval_df,
    force_http=True,
)

To also score answer correctness in code, join the retrieved context onto the LLM (or root) span by context.trace_id so each row has the question, context, and answer together, then run a correctness classifier with a {question} / {context} / {answer} template and log it the same way. For the full column conventions and the 14-day span window, see Run online evals on traces.

Step 3: Diagnose retrieval failures with embeddings

With evals in place, use Arize AX to trace bad responses back to their root cause. These embedding-based diagnostics are most useful at production scale, comparing your live queries against your full document corpus. Here’s how Arize AX surfaces each of the three failure modes introduced earlier.

Issue 1: Bad response. Arize AX automatically surfaces clusters of traces with low scores, using the retrieval relevance and correctness evals you logged in Step 2. You can also add user feedback on spans as another signal. A bad response is usually a symptom of one of the retrieval failures below.

Issue 2: Missing context. The retriever couldn’t find documents close enough to the query, meaning users ask about topics missing from your corpus. Visualize query density to find topics you need to add documentation for: set your production data as user queries and your corpus as the baseline, then look for clusters of query embeddings with no nearby context embeddings.

Issue 3: Most similar is not most relevant. A retrieved document had the closest embedding but wasn’t the most relevant. Arize AX uncovers this with LLM-assisted relevance ranking (precision@k), sending the query and retrieved context to an LLM to score relevance.

Step 4: Improve retrieval with experiments

Once you can measure quality, experiment to improve it. Generate a benchmark question set, then compare retrieval settings (chunk size, chunk overlap, and the number of documents retrieved, k) as experiments in Arize AX.

First, generate a synthetic set of test questions from the essay and put them in a dataframe with an input column:

import pandas as pd
from llama_index.llms.openai import OpenAI

GEN_TEMPLATE = """
You are an assistant that generates Q&A questions about the content below.
The questions should involve specific facts and figures, names, and elements of the text.
Do not ask any questions where the answer is not in the content.
Respond with one question per line. Do not include numbering. Generate 10 unique questions.

[START CONTENT]
{content}
[END CONTENT]
"""

with open("data/paul_graham_essay.txt") as f:
    essay = f.read()

resp = OpenAI(model="gpt-4.1").complete(GEN_TEMPLATE.format(content=essay))
questions = [q for q in resp.text.strip().split("\n") if q.strip()]
questions_df = pd.DataFrame(questions, columns=["input"])

Define a helper that builds a query engine for a given configuration, then wrap it as an experiment task. The task answers one question per dataset row and returns the answer along with the retrieved context. The context is carried through only so the evaluators can score it.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.node_parser import SentenceSplitter

documents = SimpleDirectoryReader("data").load_data()

def build_query_engine(chunk_size, chunk_overlap, k):
    splitter = SentenceSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
    index = VectorStoreIndex.from_documents(documents, transformations=[splitter])
    return index.as_query_engine(similarity_top_k=k, llm=OpenAI(model="gpt-4.1-mini"))

def make_task(chunk_size, chunk_overlap, k):
    engine = build_query_engine(chunk_size, chunk_overlap, k)

    def task(dataset_row):
        response = engine.query(dataset_row["input"])
        return {
            "output": str(response),
            "reference": "\n".join(n.text for n in response.source_nodes),
        }

    return task

Define two Phoenix Evals classifiers and wrap each as an experiment evaluator that returns an EvaluationResult with a label, score, and explanation. An evaluator receives the task’s output and the dataset_row, so it can read the question, retrieved context, and answer:

from phoenix.evals import LLM, create_classifier
from arize.experiments import EvaluationResult

llm = LLM(provider="openai", model="gpt-4.1")

RELEVANCE_TEMPLATE = """You are comparing a reference text to a question and trying to determine
if the reference text contains information relevant to answering the question. Here is the data:
    [BEGIN DATA]
    ************
    [Question]: {input}
    ************
    [Reference text]: {reference}
    [END DATA]
Is the Reference text relevant to answering the Question? Answer "relevant" or "unrelated"."""

CORRECTNESS_TEMPLATE = """You are given a question, an answer, and reference text. Determine whether
the answer correctly answers the question based on the reference text. Here is the data:
    [BEGIN DATA]
    ************
    [Question]: {input}
    ************
    [Reference]: {reference}
    ************
    [Answer]: {output}
    [END DATA]
Answer "correct" or "incorrect"."""

relevance_classifier = create_classifier(
    name="relevance", llm=llm, prompt_template=RELEVANCE_TEMPLATE,
    choices={"relevant": 1.0, "unrelated": 0.0},
)
correctness_classifier = create_classifier(
    name="correctness", llm=llm, prompt_template=CORRECTNESS_TEMPLATE,
    choices={"correct": 1.0, "incorrect": 0.0},
)

def relevance(output, dataset_row):
    score = relevance_classifier.evaluate({
        "input": dataset_row["input"],
        "reference": output["reference"],
    })[0]
    return EvaluationResult(score=score.score, label=score.label, explanation=score.explanation)

def correctness(output, dataset_row):
    score = correctness_classifier.evaluate({
        "input": dataset_row["input"],
        "reference": output["reference"],
        "output": output["output"],
    })[0]
    return EvaluationResult(score=score.score, label=score.label, explanation=score.explanation)

Create a dataset from the questions, then run one experiment per configuration. client.experiments.run executes the task across every example, applies the evaluators, and logs the results to Arize AX so you can compare how chunk size, overlap, and k affect retrieval quality in the UI:

import os
from arize import ArizeClient

client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
space_id = os.environ["ARIZE_SPACE_ID"]

dataset_name = "rag-experiments"
client.datasets.create(space=space_id, name=dataset_name, examples=questions_df)

def run_experiment(name, chunk_size, chunk_overlap, k):
    client.experiments.run(
        name=name,
        dataset=dataset_name,
        space=space_id,
        task=make_task(chunk_size, chunk_overlap, k),
        evaluators=[relevance, correctness],
    )

# Compare retrieval settings
run_experiment("k2-chunk200", chunk_size=200, chunk_overlap=10, k=2)
run_experiment("k4-chunk200", chunk_size=200, chunk_overlap=10, k=4)
run_experiment("k2-chunk1000", chunk_size=1000, chunk_overlap=50, k=2)

For the full datasets and experiments API, see Run an experiment and Log an experiment.

Summary

In this tutorial, you built a RAG application and used Arize AX to:

• Trace retrieval, embeddings, and LLM calls end to end
• Score retrieval relevance and answer correctness with LLM-as-a-judge
• Diagnose missing-context and irrelevant-retrieval failures using embeddings
• Run experiments to improve retrieval across chunk size, overlap, and k

Next steps

• Manually create retriever spans that log your retriever inputs and outputs
• Evaluate across more dimensions such as hallucination, citation, and user frustration with prebuilt evaluators
• Go deeper on datasets and experiments