> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Research

> Collection of advanced experiments and benchmarks in LLM evaluation, instrumentation, and agent systems

<CardGroup>
  <Card icon="clock" href="#time-series-evals-with-openai-o1-preview" title="Time Series Evals with OpenAI o1-preview" horizontal />

  <Card icon="brain" href="#prompt-caching-benchmarking" title="Prompt Caching Benchmarking" horizontal />

  <Card icon="robot" href="#multi-agent-systems-swarm" title="Multi-Agent Systems: Swarm" horizontal />

  <Card icon="microchip" href="#instrumenting-llms-with-otel" title="Instrumenting LLMs with OTel" horizontal />

  <Card icon="robot" href="#comparing-agent-frameworks" title="Comparing Agent Frameworks" horizontal />

  <Card icon="code" href="#testing-generation-in-rag" title="Testing Generation in RAG" horizontal />
</CardGroup>

## Time Series Evals with OpenAI o1-preview

<Frame>
  <iframe width="100%" height="420" src="https://cdn.iframe.ly/wo1LxVY7" title="LLM Evals and LLM as a Judge: Fundamentals" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen />
</Frame>

We benchmarked o1-preview on our hardest eval task - **time series trend evaluations**. This post compares that performance against GPT-4o-mini, Claude 3.5 sonnet, and GPT-4o.

<CardGroup>
  <Card title="o1-preview Time Series EvaluationsArize AI" href="https://arize.com/blog/o1-preview-time-series-evaluations" cta="Arize AI" icon="clock" />
</CardGroup>

## Prompt Caching Benchmarking

<Frame>
  <iframe width="100%" height="420" src="https://cdn.iframe.ly/Qea8uRSv" title="LLM Evals and LLM as a Judge: Fundamentals" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen />
</Frame>

We compare the performance and cost savings of **prompt caching on Anthropic vs OpenAI**.

<CardGroup>
  <Card title="How to Make Your AI App Feel Magical: Prompt CachingArize AI" href="https://arize.com/blog/prompt-caching-analysis" cta="Arize AI" icon="brain" />
</CardGroup>

## Multi-Agent Systems: Swarm

<Frame>
  <iframe width="100%" height="420" src="https://cdn.iframe.ly/IUEEfxxN" title="LLM Evals and LLM as a Judge: Fundamentals" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen />
</Frame>

We compare and contrast **OpenAI's experimental Swarm repo** against other popular multi-agent frameworks: **Autogen and CrewAI**

<CardGroup>
  <Card title="Comparing OpenAI Swarm with other Multi Agent FrameworksArize AI" href="https://arize.com/blog/comparing-openai-swarm" cta="Arize AI" icon="robot" />
</CardGroup>

## Instrumenting LLMs with OTel

<Frame>
  <iframe width="100%" height="420" src="https://cdn.iframe.ly/rmqkMGSE" title="LLM Evals and LLM as a Judge: Fundamentals" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen />
</Frame>

Lessons learned from our journey to one million downloads of our OpenTelemetry wrapper, [OpenInference](https://github.com/Arize-ai/openinference).

<CardGroup>
  <Card title="Zero to a Million: Instrumenting LLMs with OTELArize AI" href="https://arize.com/blog/zero-to-a-million-instrumenting-llms-with-otel" cta="Arize AI" icon="robot" />
</CardGroup>

## Comparing Agent Frameworks

<Frame>
  <iframe width="100%" height="420" src="https://cdn.iframe.ly/AUHGLGQQ" title="LLM Evals and LLM as a Judge: Fundamentals" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen />
</Frame>

We built the [same agent](https://github.com/Arize-ai/phoenix/tree/main/examples/agent_framework_comparison) in LangGraph, LlamaIndex Workflows, CrewAI, Autogen, and pure code. See how each framework compares.

<CardGroup>
  <Card title="Comparing Agent FrameworksArize AI" href="https://arize.com/blog-course/llm-agent-how-to-set-up/comparing-agent-frameworks" cta="Arize AI" icon="microchip" />
</CardGroup>

## Testing Generation in RAG

<Frame>
  <iframe width="100%" height="420" src="https://cdn.iframe.ly/ij3ahLhV" title="LLM Evals and LLM as a Judge: Fundamentals" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen />
</Frame>

Testing the generation stage of RAG across GPT-4 and Claude 2.1.

<CardGroup>
  <Card title="Evaluating the Generation Stage in RAGArize AI" href="https://arize.com/blog/evaluating-the-generation-stage-in-rag" cta="Arize AI" icon="code" />
</CardGroup>