Instrument LiteLLM agents

Trace an agent built with LiteLLM in Arize AX: auto-instrument completion calls, add manual agent and tool spans, and trace the LiteLLM Proxy gateway.

LiteLLM is an open-source library that gives you a single, unified interface to call 100+ LLMs using a unified format. Learn how to instrument agents built on LiteLLM to capture every model call, tool call, and agent step so you can observe, evaluate, and improve quality. This guide auto-instruments the LiteLLM Python SDK with the OpenInference LiteLLM instrumentor and enables the LiteLLM Proxy’s native Arize callback to send traces to Arize AX.

Overview

In this guide, you will auto-instrument the LiteLLM SDK, add manual spans around your agent’s loop and tools, and read the resulting trace tree in Arize AX. Then, you will route the LiteLLM Proxy to Arize AX to capture every model call that flows through the gateway.

The two layers work together. The instrumentor captures each model call automatically, while the loop and the tool executions are your own code, so you wrap those in manual spans. Combined, they produce one trace that shows what the agent decided and what each tool returned, not just the raw model calls.

This guide assumes familiarity with:

• Python
• The LiteLLM completion API and the tool-use loop (completion, tool_calls, tool results)
• Environment variables and package installation

By the end of this guide, you will be able to:

• Auto-instrument LiteLLM so each completion() call is captured as an LLM span
• Add manual AGENT and TOOL spans so the loop and tool executions appear in the trace
• Group traces by conversation using session IDs
• Trace the LiteLLM Proxy gateway with its native Arize AX callback

Before you start

You need:

• An existing agent built with LiteLLM
• An Arize AX account (sign up)
• A key for one of LiteLLM’s supported providers. LiteLLM routes to the provider named in the model string, so anthropic/... reads ANTHROPIC_API_KEY, openai/... reads OPENAI_API_KEY, and so on.

Get your Arize AX credentials

Sign in to your Arize AX account and create a tracing project from Projects → New Tracing Project. The setup page shows the credentials you need: copy your Space ID, then click Create API Key to generate a key and save it somewhere safe. You set them as ARIZE_SPACE_ID and ARIZE_API_KEY when you configure your environment below, and they route your traces to the correct space.

Trace an agent built with the LiteLLM SDK

Prerequisites: Python 3.10 or later.

The steps below instrument a small example agent, a health coach with a single tool, so the trace shows the AGENT → LLM → TOOL shape end to end. Attach the instrumentor once at startup, then wrap your agent loop and each tool call in manual spans.

1. Install the dependencies

   Install the LiteLLM instrumentor, the OpenInference helper used for session grouping, and LiteLLM itself.

   pip install litellm \
     openinference-instrumentation-litellm \
     openinference-instrumentation \
     arize-otel \
     "wrapt<2"
   
2. Configure credentials

   Set your Arize AX and provider credentials as environment variables. The tracing setup reads them at startup, so you keep secrets out of your source.

   export ARIZE_SPACE_ID="<your-space-id>"
   export ARIZE_API_KEY="<your-api-key>"
   export ARIZE_PROJECT_NAME="litellm-health-coach"
   export ANTHROPIC_API_KEY="<your-anthropic-api-key>"
   

   ARIZE_PROJECT_NAME is the project your traces land in. Name it for the app so runs stay grouped where you expect them. Set the provider key that matches the model string you call.

3. Auto-instrument LiteLLM

   Set up tracing in a dedicated module so it runs once, before your agent calls LiteLLM. This registers a tracer provider with your Arize credentials and attaches the LiteLLM instrumentor, which captures every completion() call as an LLM span.

   instrumentation.py

   import os
   
   from arize.otel import register
   from openinference.instrumentation.litellm import LiteLLMInstrumentor
   
   # register() reads ARIZE_SPACE_ID and ARIZE_API_KEY from the environment
   # when the arguments are omitted. Pass them explicitly if you prefer.
   tracer_provider = register(
       space_id=os.environ["ARIZE_SPACE_ID"],
       api_key=os.environ["ARIZE_API_KEY"],
       project_name=os.environ.get("ARIZE_PROJECT_NAME", "litellm-health-coach"),
   )
   
   # Auto-instrument LiteLLM: each completion() call becomes an LLM span.
   LiteLLMInstrumentor().instrument(tracer_provider=tracer_provider)
   
4. Trace the agent loop

   This is an ordinary LiteLLM tool-use loop with tracing added around it. The tracing is the two tracer.start_as_current_span(...) blocks and their attributes; the completion() call and the tool dispatch are the code you already have. Wrapping the loop in an AGENT span and each tool execution in a TOOL span is the second of the two instrumentation moves. The completion() calls run inside the AGENT span, so the instrumentor’s LLM spans nest underneath it automatically. The OpenInference attributes are set as string keys ("openinference.span.kind", "input.value").

   import json
   
   import instrumentation  # noqa: F401 - runs the tracing setup first
   import litellm
   from opentelemetry import trace
   from opentelemetry.trace import Status, StatusCode
   
   tracer = trace.get_tracer(__name__)
   
   # LiteLLM routes on the model string. Swap the provider prefix and key to
   # call OpenAI, Bedrock, Vertex AI, Groq, and more with the same code.
   MODEL = "anthropic/claude-sonnet-4-6"
   
   TOOLS = [
       {
           "type": "function",
           "function": {
               "name": "get_daily_calories",
               "description": "Recommended daily calorie intake for an age and activity level.",
               "parameters": {
                   "type": "object",
                   "properties": {
                       "age": {"type": "integer"},
                       "activity_level": {
                           "type": "string",
                           "enum": ["low", "moderate", "high"],
                       },
                   },
                   "required": ["age", "activity_level"],
               },
           },
       }
   ]
   
   
   def run_tool(name: str, args: dict) -> str:
       # Replace with your real tool implementation.
       if name == "get_daily_calories":
           base = {"low": 1800, "moderate": 2200, "high": 2600}[args["activity_level"]]
           return json.dumps({"recommended_calories": base})
       return json.dumps({"error": f"unknown tool {name}"})
   
   
   def run_agent(user_input: str) -> str:
       with tracer.start_as_current_span("health-coach") as agent_span:
           agent_span.set_attribute("openinference.span.kind", "AGENT")
           agent_span.set_attribute("input.value", user_input)
   
           messages = [{"role": "user", "content": user_input}]
   
           while True:
               # Auto-instrumented: each call appears as a child LLM span.
               response = litellm.completion(
                   model=MODEL,
                   messages=messages,
                   tools=TOOLS,
               )
               message = response.choices[0].message
               messages.append(message.model_dump())
   
               tool_calls = message.tool_calls or []
               if not tool_calls:
                   break
   
               for call in tool_calls:
                   name = call.function.name
                   args = json.loads(call.function.arguments)
                   # Manual TOOL span for each tool execution.
                   with tracer.start_as_current_span(name) as tool_span:
                       tool_span.set_attribute("openinference.span.kind", "TOOL")
                       tool_span.set_attribute("tool.name", name)
                       tool_span.set_attribute("input.value", json.dumps(args))
                       result = run_tool(name, args)
                       tool_span.set_attribute("output.value", result)
                   messages.append(
                       {
                           "role": "tool",
                           "tool_call_id": call.id,
                           "content": result,
                       }
                   )
   
           final_text = message.content or ""
           agent_span.set_attribute("output.value", final_text)
           agent_span.set_status(Status(StatusCode.OK))
           return final_text
   
   
   print(run_agent("I'm 34 and moderately active. What's my daily calorie target?"))
   
   # Short-lived script: flush batched spans before the process exits.
   instrumentation.tracer_provider.force_flush()
   

   In your own app, call run_agent from wherever you handle input. The last two lines run it as a one-off script; force_flush() matters only for short-lived processes, since long-running services flush on their own.

5. Group traces by conversation

   To follow a multi-turn conversation as a single thread, tag its runs with a shared session ID that is stable for the conversation, such as the user or thread ID. Wrap the existing run_agent call in the using_session() context manager, one line, and every span emitted inside, manual and auto, carries the same session.id.

   from openinference.instrumentation import using_session
   
   with using_session(session_id):
       run_agent(user_input)
   

   Arize AX groups the tagged runs together, so you can replay a full conversation.

6. Verify in Arize AX

   Open your Arize AX space and select the project you set in ARIZE_PROJECT_NAME. Within about 30 seconds of a run, a new trace appears with this shape:

   • An AGENT root span (health-coach) carrying the user input and final answer.
   • One or more LLM child spans, one per completion() call, with the prompt, response, model name, and token counts filled in automatically by the LiteLLM instrumentor.
   • A TOOL child span for each tool the agent ran, with the tool inputs and outputs.

   Without the manual spans you would see only the LLM calls. The AGENT and TOOL spans are what turn a set of model calls into a readable agent trace.

   If no traces appear, see Troubleshooting.

Trace the LiteLLM Proxy

The LiteLLM Proxy is a standalone gateway that exposes an OpenAI-compatible API and routes requests to any provider. Because clients talk to the gateway over HTTP, the in-process instrumentor never sees those calls. Instead, the Proxy exports to Arize AX through its own native callback, so every request routed through the gateway is captured as an LLM span regardless of which language or SDK the client uses.

1. Install the Proxy

   Install the gateway alongside the OpenTelemetry packages the arize callback exports through.

   pip install "litellm[proxy]" \
     opentelemetry-sdk \
     opentelemetry-exporter-otlp-proto-grpc
   

   Install both: the gateway does not pull in the OpenTelemetry SDK on its own, and without it the callback fails to start and no traces are sent.

2. Enable the Arize callback

   Register arize as a callback in your gateway configuration. The Proxy reads your Arize credentials from the environment and exports a span for every routed request.

   config.yaml

   model_list:
     - model_name: claude-sonnet-4-6
       litellm_params:
         model: anthropic/claude-sonnet-4-6
   
   litellm_settings:
     callbacks: ["arize"]
   
3. Configure credentials

   Set your Arize AX and provider credentials in the environment the gateway process runs in. The arize callback reads them at startup.

   export ARIZE_SPACE_ID="<your-space-id>"
   export ARIZE_API_KEY="<your-api-key>"
   export ARIZE_PROJECT_NAME="litellm-proxy"
   export ANTHROPIC_API_KEY="<your-anthropic-api-key>"
   

   The callback reads ARIZE_SPACE_ID, ARIZE_API_KEY, and ARIZE_PROJECT_NAME, then exports spans to Arize AX over OTLP.

4. Run the gateway

   Start the gateway with your config using the litellm command.

   litellm --config config.yaml
   

   The gateway starts on http://localhost:4000 by default.

5. Send a request through the gateway

   Point any OpenAI-compatible client at the gateway URL and call the model name you defined in config.yaml.

   curl http://localhost:4000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "claude-sonnet-4-6",
       "messages": [{"role": "user", "content": "Why is the ocean salty?"}]
     }'
   
6. Verify in Arize AX

   Open your Arize AX space and select the project you set in ARIZE_PROJECT_NAME. Within about 30 seconds, a trace appears with an LLM span for the routed request, carrying the prompt, response, model name, and token counts.

   The gateway captures the model call. To also see AGENT and TOOL spans, add the manual spans shown in Trace an agent built with the LiteLLM SDK to the client that calls the gateway.

What gets captured

The two layers combine into one trace:

• LLM spans (automatic). The LiteLLM instrumentor records each completion() call: prompt, response, the tool calls the model emits, the model name, and token usage. The same instrumentor covers acompletion(), completion_with_retries(), embedding(), aembedding(), image_generation(), and aimage_generation().
• AGENT and TOOL spans (manual). Your spans record the loop as a whole and each tool execution, so the trace shows what the agent decided to do and what each tool returned, not just the raw model calls.
• Session grouping. The session.id you set threads related runs into one conversation.

Because the manual and auto spans share the same tracer provider, the LLM spans nest under your AGENT span automatically as long as the completion() calls run inside it.

Troubleshooting

• No traces in Arize AX. Confirm ARIZE_SPACE_ID and ARIZE_API_KEY are set in the same shell that runs your app. Enable OpenTelemetry debug logs with export OTEL_LOG_LEVEL=debug and rerun.
• LLM spans missing. LiteLLMInstrumentor().instrument(...) must run before the first completion() call. Import the tracing module first.
• LLM spans not nested under the agent span. The completion() call must run inside the active AGENT span. Keep it inside the with tracer.start_as_current_span(...) block.
• 401 from the provider. LiteLLM picks the provider from the model string (anthropic/..., openai/..., groq/...). Make sure the matching key (ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.) is set.
• A short-lived script exports nothing. Spans are batched by default, so a script that exits right after the run can quit before they flush. Call tracer_provider.force_flush() before the process exits. Long-running apps and servers flush on their own.
• wrap_function_wrapper() got an unexpected keyword argument 'module'. A wrapt 2.x release changed that function’s signature, which older instrumentor pins do not yet account for. Pin wrapt below 2 until the pins catch up: pip install "wrapt<2".
• Proxy traces missing. Confirm callbacks: ["arize"] is under litellm_settings in config.yaml, and that the gateway process has ARIZE_SPACE_ID and ARIZE_API_KEY set. The Proxy logs initializing callbacks=['arize'] on startup when the callback is registered.
• Proxy logs Error initializing custom logger: No module named 'opentelemetry'. The arize callback needs the OpenTelemetry SDK, which the litellm[proxy] extra does not install. This error is non-blocking, so requests still succeed while no traces are sent. Install opentelemetry-sdk and opentelemetry-exporter-otlp-proto-grpc as shown.

Next steps

Combine auto and manual

The general pattern for mixing auto LLM spans with manual spans.

Agent trajectory

Render the agent’s execution as an interactive graph.

Set up sessions

Group multi-turn runs into conversations with session and user IDs.

LiteLLM integration

The reference page for the LiteLLM instrumentor and provider routing.