> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Create an annotation queue
> Create a new annotation queue.
**Payload Requirements**
- The annotation queue name must be unique within the given space (among active queues).
- At least one `annotation_config_id` is required, and all configs must belong to the specified space.
- Do not include system-managed fields on input: `id`, `created_at`, `updated_at`.
- If `assignment_method` is not provided, it defaults to `"all"`.
**Valid example**
```json
{
"name": "Quality Review Queue",
"space_id": "spc_xyz789",
"annotation_config_ids": ["ac_abc123"],
"annotator_emails": ["reviewer@example.com"],
"assignment_method": "all"
}
```
**Valid example with records**
```json
{
"name": "Quality Review Queue",
"space_id": "spc_xyz789",
"annotation_config_ids": ["ac_abc123"],
"annotator_emails": ["reviewer@example.com"],
"records": [
{"record_type": "span", "project_id": "prj_abc", "start_time": "2024-01-15T00:00:00Z", "end_time": "2024-01-16T00:00:00Z", "span_ids": ["span_001"]},
{"record_type": "example", "dataset_id": "ds_xyz", "example_ids": ["ex_001", "ex_002"]}
]
}
```
**Invalid example** (missing required annotation_config_ids)
```json
{
"name": "My Queue",
"space_id": "spc_xyz789"
}
```
This endpoint is in beta, read more [here](https://arize.com/docs/ax/rest-reference#api-version-stages).
## OpenAPI
````yaml https://api.arize.com/v2/spec.yaml post /v2/annotation-queues
openapi: 3.0.3
info:
title: Arize REST API
version: 2.0.0
description: |
API specification for the backend data server. The API is hosted globally
at https://api.arize.com/v2 or in your own environment.
license:
name: Apache-2.0
url: https://www.apache.org/licenses/LICENSE-2.0
servers:
- description: Global
url: https://api.arize.com
- description: Regional
url: https://api.{region}.arize.com
variables:
region:
default: eu-west-1a
enum:
- eu-west-1a
- ca-central-1a
- description: Custom Host
url: https://{host}
variables:
host:
default: api.arize.com
security:
- bearerAuth: []
tags:
- name: AI Integrations
description: |
AI integrations configure access to external LLM providers (e.g. OpenAI,
Azure OpenAI, AWS Bedrock, Vertex AI). Integrations can be scoped to the
entire account, a specific organization, or a specific space.
- name: Annotation Configs
description: >
Annotation configs allow you to define consistent annotation schemas that
can be reused across your workspace, ensuring evaluations are structured
and
comparable over time.
- name: Annotation Queues
description: >
Annotation queues help you organize and manage human evaluation workflows.
Use queues to assign spans or examples to annotators for review and
labeling.
- name: API Keys
description: >
API keys are used to authenticate requests to the Arize API. List your
keys
to view metadata; the raw secret is never returned after creation.
- name: Datasets
description: |
Datasets are structured, version-controlled example collections you use to
run, evaluate, and track LLM experiments.
- name: Evaluators
description: >
Evaluators are reusable evaluation configurations used to assess the
quality
of LLM outputs. They can be template-based (using LLM judges) or
code-based.
- name: Experiments
description: >
Experiments let you systematically test prompt/model changes using
datasets,
tasks, and evaluators.
- name: Integrations
description: >
Integrations configure access to external LLM providers (e.g. OpenAI,
Azure OpenAI, AWS Bedrock, Vertex AI), notifications services (e.g.
PagerDuty, Slack), and
your own agents. Integrations can be scoped to the entire account, a
specific
organization, or a specific space.
- name: Organizations
description: >
Organizations are top-level containers within an Arize AX account for
grouping spaces.
- name: Projects
description: |
Projects represent LLM applications being monitored in Arize where you can
observe traces and spans.
- name: Prompts
description: >
Prompts are reusable, versioned templates for LLM interactions. Use
prompts
to standardize and manage how you interact with LLMs across your
application.
- name: Resource Restrictions
description: |
Endpoints for restricting and unrestricting resources (projects, models).
- name: Role Bindings
description: |
Role bindings assign a role to a user on a resource. REST currently
supports space- and project-scoped bindings.
- name: Roles
description: >
Roles define sets of permissions that can be assigned to users within an
account. Create custom roles to tailor access control to your team's
needs.
- name: Spaces
description: >
Spaces are containers within an organization for grouping related
projects,
datasets, and experiments, enabling collaboration or isolated
experimentation
with role-based access control.
- name: Spans
description: |
Spans represent individual operations within a trace. A span captures the
timing, status, and attributes of a single operation in your application.
- name: Tasks
description: |
Tasks are configurable units of work that tie one or more evaluators to a
data source (project or dataset). Use tasks to automate evaluation of LLM
outputs, with support for continuous evaluation and backfill runs.
- name: Users
description: >
Users represent members of an account. The Users endpoints allow creating,
listing, updating (display name), and removing users from the account
programmatically.
paths:
/v2/annotation-queues:
post:
tags:
- Annotation Queues
summary: Create an annotation queue
description: >
Create a new annotation queue.
**Payload Requirements**
- The annotation queue name must be unique within the given space (among
active queues).
- At least one `annotation_config_id` is required, and all configs must
belong to the specified space.
- Do not include system-managed fields on input: `id`, `created_at`,
`updated_at`.
- If `assignment_method` is not provided, it defaults to `"all"`.
**Valid example**
```json
{
"name": "Quality Review Queue",
"space_id": "spc_xyz789",
"annotation_config_ids": ["ac_abc123"],
"annotator_emails": ["reviewer@example.com"],
"assignment_method": "all"
}
```
**Valid example with records**
```json
{
"name": "Quality Review Queue",
"space_id": "spc_xyz789",
"annotation_config_ids": ["ac_abc123"],
"annotator_emails": ["reviewer@example.com"],
"records": [
{"record_type": "span", "project_id": "prj_abc", "start_time": "2024-01-15T00:00:00Z", "end_time": "2024-01-16T00:00:00Z", "span_ids": ["span_001"]},
{"record_type": "example", "dataset_id": "ds_xyz", "example_ids": ["ex_001", "ex_002"]}
]
}
```
**Invalid example** (missing required annotation_config_ids)
```json
{
"name": "My Queue",
"space_id": "spc_xyz789"
}
```
This endpoint is in beta, read more
[here](https://arize.com/docs/ax/rest-reference#api-version-stages).
operationId: annotation_queues_create
requestBody:
$ref: '#/components/requestBodies/CreateAnnotationQueueRequestBody'
responses:
'201':
$ref: '#/components/responses/AnnotationQueueCreate'
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'403':
$ref: '#/components/responses/Forbidden'
'404':
$ref: '#/components/responses/NotFound'
'409':
$ref: '#/components/responses/Conflict'
'422':
$ref: '#/components/responses/UnprocessableEntity'
'429':
$ref: '#/components/responses/RateLimitExceeded'
components:
requestBodies:
CreateAnnotationQueueRequestBody:
description: Body containing annotation queue creation parameters
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/CreateAnnotationQueueRequestBody'
example:
name: Quality Review Queue
space_id: spc_xyz789
annotation_config_ids:
- ac_abc123
annotator_emails:
- annotator1@example.com
assignment_method: all
responses:
AnnotationQueueCreate:
description: Returns the created annotation queue
content:
application/json:
schema:
$ref: '#/components/schemas/AnnotationQueue'
example:
id: aq_abc123
name: Quality Review Queue
space_id: spc_xyz789
instructions: Review each response for accuracy and helpfulness
annotation_configs:
- id: ac_001
name: accuracy
type: continuous
space_id: spc_xyz789
minimum_score: 0
maximum_score: 10
optimization_direction: maximize
created_at: '2024-01-10T08:00:00Z'
- id: ac_002
name: sentiment
type: categorical
space_id: spc_xyz789
values:
- label: positive
score: 1
- label: neutral
score: 0
- label: negative
score: -1
optimization_direction: maximize
created_at: '2024-01-10T08:00:00Z'
annotators:
- id: usr_001
email: reviewer@example.com
- id: usr_002
email: annotator@example.com
created_at: '2024-01-15T10:30:00Z'
updated_at: '2024-01-15T10:30:00Z'
BadRequest:
description: Invalid request
content:
application/problem+json:
schema:
$ref: '#/components/schemas/Problem'
example:
status: 400
title: Invalid request parameters
detail: The 'name' field is required and must be a non-empty string.
instance: /resource
type: https://arize.com/docs/ax/rest-reference/errors#invalid-request
Unauthorized:
description: Authentication is required
content:
application/problem+json:
schema:
$ref: '#/components/schemas/Problem'
example:
status: 401
title: Authentication required
detail: You must be authenticated to access this resource.
instance: /resource
type: >-
https://arize.com/docs/ax/rest-reference/errors#authentication-required
Forbidden:
description: Insufficient permissions to access this resource
content:
application/problem+json:
schema:
$ref: '#/components/schemas/Problem'
example:
status: 403
title: Access forbidden
detail: You do not have permission to access this resource.
instance: /resource/12345
type: https://arize.com/docs/ax/rest-reference/errors#access-forbidden
NotFound:
description: Not found
content:
application/problem+json:
schema:
$ref: '#/components/schemas/Problem'
example:
status: 404
title: Resource not found
detail: The requested resource with ID '12345' was not found.
instance: /resource/12345
type: https://arize.com/docs/ax/rest-reference/errors#resource-not-found
Conflict:
description: Resource conflict
content:
application/problem+json:
schema:
$ref: '#/components/schemas/Problem'
example:
status: 409
title: Resource conflict
detail: A resource with the given identifier already exists.
instance: /resource
type: https://arize.com/docs/ax/rest-reference/errors#resource-conflict
UnprocessableEntity:
description: Unprocessable entity
content:
application/problem+json:
schema:
$ref: '#/components/schemas/Problem'
example:
status: 422
title: Unprocessable Entity
detail: Minimum score must be less than maximum score.
instance: /annotation-configs
type: >-
https://arize.com/docs/ax/rest-reference/errors#unprocessable-entity
RateLimitExceeded:
description: Rate limit exceeded
headers:
Retry-After:
description: |
When throttled (429), how long to wait before retrying. Value is
either a delta-seconds integer.
schema:
type: integer
minimum: 0
example: 42
content:
application/problem+json:
schema:
$ref: '#/components/schemas/Problem'
example:
status: 429
title: Rate limit exceeded
detail: >-
You have exceeded the allowed number of requests. Please try again
later.
instance: /resource
type: >-
https://arize.com/docs/ax/rest-reference/errors#rate-limit-exceeded
schemas:
CreateAnnotationQueueRequestBody:
type: object
required:
- name
- space_id
- annotation_config_ids
- annotator_emails
properties:
name:
type: string
maxLength: 255
description: >-
The name of the annotation queue. Must be unique within the space
for active queues.
example: Quality Review Queue
space_id:
type: string
description: The space ID that the annotation queue belongs to
example: spc_xyz789
instructions:
type: string
maxLength: 5000
description: Instructions for annotators working on this queue
example: Review each response for accuracy and helpfulness
annotation_config_ids:
type: array
items:
type: string
minItems: 1
description: >-
IDs of annotation configs to associate with this queue. All configs
must belong to the same space.
example:
- ac_abc123
- ac_def456
annotator_emails:
type: array
items:
$ref: '#/components/schemas/Email'
minItems: 1
description: >-
Email addresses of annotators to assign to the queue. Emails are
resolved to user IDs server-side.
example:
- annotator1@example.com
- annotator2@example.com
assignment_method:
allOf:
- $ref: '#/components/schemas/AssignmentMethod'
default: all
description: >-
How records are assigned to annotators. Defaults to `all` when
omitted.
record_sources:
type: array
maxItems: 2
items:
$ref: '#/components/schemas/AnnotationQueueRecordInput'
description: >-
Record sources to add to the annotation queue on creation. At most 2
record sources (projects or datasets) may be provided in a single
create request. Additional records from other sources can be added
after creation.
additionalProperties: false
AnnotationQueue:
type: object
required:
- id
- name
- space_id
- annotators
- created_at
- updated_at
properties:
id:
type: string
description: The unique identifier for the annotation queue
example: aq_abc123
name:
type: string
description: The name of the annotation queue
example: Quality Review Queue
space_id:
type: string
description: The space id the annotation queue belongs to
example: spc_xyz789
instructions:
type: string
nullable: true
description: The instructions for the annotation queue
example: Review each response for accuracy and helpfulness
annotation_configs:
type: array
items:
$ref: '#/components/schemas/AnnotationConfig'
description: The annotation configs associated with this queue
annotators:
type: array
items:
$ref: '#/components/schemas/AnnotatorUser'
description: Users assigned as annotators to this queue
created_at:
type: string
format: date-time
description: The timestamp for when the annotation queue was created
example: '2024-01-15T10:30:00Z'
updated_at:
type: string
format: date-time
description: The timestamp for when the annotation queue was last updated
example: '2024-01-20T14:45:00Z'
additionalProperties: false
Problem:
type: object
description: RFC 9457 Problem Details
properties:
title:
type: string
description: A short, human-readable summary of the problem type
status:
type: integer
description: >-
The HTTP status code generated by the origin server for this
occurrence of the problem
type:
type: string
format: uri-reference
description: A URI reference that identifies the problem type
detail:
type: string
description: >-
A human-readable explanation specific to this occurrence of the
problem
instance:
type: string
format: uri-reference
description: >-
A URI reference that identifies the specific occurrence of the
problem
required:
- title
- status
additionalProperties: false
Email:
type: string
format: email
description: An email address
example: user@example.com
AssignmentMethod:
type: string
enum:
- all
- random
default: all
description: |
How records are assigned to annotators. Defaults to "all".
- `all`: Every annotator is assigned to every record.
- `random`: Each record is randomly assigned to one annotator.
AnnotationQueueRecordInput:
oneOf:
- $ref: '#/components/schemas/AnnotationQueueExampleRecordInput'
- $ref: '#/components/schemas/AnnotationQueueSpanRecordInput'
- $ref: '#/components/schemas/AnnotationQueueTraceRecordInput'
discriminator:
propertyName: record_type
mapping:
example:
$ref: '#/components/schemas/AnnotationQueueExampleRecordInput'
span:
$ref: '#/components/schemas/AnnotationQueueSpanRecordInput'
trace:
$ref: '#/components/schemas/AnnotationQueueTraceRecordInput'
AnnotationConfig:
type: object
additionalProperties: true
discriminator:
propertyName: type
mapping:
continuous:
$ref: '#/components/schemas/ContinuousAnnotationConfig'
categorical:
$ref: '#/components/schemas/CategoricalAnnotationConfig'
freeform:
$ref: '#/components/schemas/FreeformAnnotationConfig'
oneOf:
- $ref: '#/components/schemas/ContinuousAnnotationConfig'
- $ref: '#/components/schemas/CategoricalAnnotationConfig'
- $ref: '#/components/schemas/FreeformAnnotationConfig'
AnnotatorUser:
type: object
description: A user assigned as an annotator, identified by ID and email.
required:
- id
- email
properties:
id:
type: string
description: The unique identifier for the user
email:
$ref: '#/components/schemas/Email'
additionalProperties: false
AnnotationQueueExampleRecordInput:
type: object
required:
- record_type
- dataset_id
properties:
record_type:
type: string
enum:
- example
description: >-
Discriminator identifying this record source as dataset examples.
Must be `example` for dataset example records.
dataset_id:
type: string
description: The dataset ID these examples belong to
dataset_version_id:
type: string
description: >
Optional. The specific dataset version to use. If omitted, the
latest version is used.
example_ids:
type: array
items:
type: string
description: >
Optional. List of example IDs within the dataset to add to the
queue. If omitted, all examples in the dataset (or dataset version)
are added.
additionalProperties: false
AnnotationQueueSpanRecordInput:
type: object
required:
- record_type
- project_id
- span_ids
- start_time
- end_time
properties:
record_type:
type: string
enum:
- span
description: >-
Discriminator identifying this record source as project spans. Must
be `span` for span records.
project_id:
type: string
description: The project ID these spans belong to
start_time:
type: string
format: date-time
description: >
Start of the time range to search for spans in Druid. The range
(end_time - start_time) must not exceed 7 days.
end_time:
type: string
format: date-time
description: |
End of the time range. Must be after start_time.
span_ids:
type: array
items:
type: string
minItems: 1
description: List of span IDs to add to the queue
additionalProperties: false
AnnotationQueueTraceRecordInput:
type: object
required:
- record_type
- project_id
- trace_ids
- start_time
- end_time
properties:
record_type:
type: string
enum:
- trace
description: Discriminator identifying this record as a trace record.
project_id:
type: string
description: The project ID these traces belong to.
start_time:
type: string
format: date-time
description: >
Start of the time range used to resolve each trace's root span. The
range (end_time - start_time) must not exceed 7 days.
end_time:
type: string
format: date-time
description: |
End of the time range. Must be after start_time.
trace_ids:
type: array
items:
type: string
minItems: 1
description: |
List of trace IDs to add to the queue.
additionalProperties: false
ContinuousAnnotationConfig:
allOf:
- $ref: '#/components/schemas/AnnotationConfigBase'
- type: object
required:
- type
- minimum_score
- maximum_score
properties:
type:
type: string
enum:
- continuous
description: Discriminator value identifying a continuous annotation config.
minimum_score:
type: number
format: double
description: The minimum score value
maximum_score:
type: number
format: double
description: The maximum score value
optimization_direction:
allOf:
- $ref: '#/components/schemas/OptimizationDirection'
description: >-
Direction for optimization. When absent, the server treats the
value as `none`.
CategoricalAnnotationConfig:
allOf:
- $ref: '#/components/schemas/AnnotationConfigBase'
- type: object
required:
- type
- values
properties:
type:
type: string
enum:
- categorical
description: Discriminator value identifying a categorical annotation config.
values:
type: array
description: An array of categorical annotation values
items:
$ref: '#/components/schemas/CategoricalAnnotationValue'
optimization_direction:
allOf:
- $ref: '#/components/schemas/OptimizationDirection'
description: >-
Direction for optimization. When absent, the server treats the
value as `none`.
FreeformAnnotationConfig:
allOf:
- $ref: '#/components/schemas/AnnotationConfigBase'
- type: object
required:
- type
properties:
type:
type: string
enum:
- freeform
description: Discriminator value identifying a freeform annotation config.
AnnotationConfigBase:
type: object
required:
- id
- name
- created_at
- space_id
properties:
id:
type: string
description: The unique identifier for the annotation config
name:
type: string
description: The name of the annotation config
created_at:
type: string
description: The timestamp for when the annotation config was created
format: date-time
space_id:
type: string
description: The space id the annotation config belongs to
OptimizationDirection:
type: string
enum:
- maximize
- minimize
- none
default: none
description: |
The direction for optimization. Defaults to `none` when omitted.
- maximize: higher scores are better
- minimize: lower scores are better
- none: higher or lower scores are neither better nor worse
CategoricalAnnotationValue:
type: object
required:
- label
properties:
label:
type: string
description: The label value
score:
type: number
format: double
description: A score to associate with the label
additionalProperties: false
securitySchemes:
bearerAuth:
type: http
scheme: bearer
bearerFormat:
description: >
Most Arize AI endpoints require authentication. For those endpoints that
require authentication, include your API key in the request header using
the format
``` Authorization: Bearer
```
````