Back to Details

arize-annotation

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Arize Annotation Skill

Arize Annotation Skill

This skill focuses on annotation configs — the schema for human feedback — and on programmatically annotating project spans via the Python SDK. Human review in the Arize UI (including annotation queues, datasets, and experiments) still depends on these configs; there is no 
ax
CLI for queues yet.
Direction: Human labeling in Arize attaches values defined by configs to spans, dataset examples, experiment-related records, and queue items in the product UI. What is documented here: 
ax annotation-configs
and bulk span updates with 
ArizeClient.spans.update_annotations
.
本Skill专注于标注配置（annotation configs）——即人工反馈的Schema——以及通过Python SDK以编程方式标注项目Span。Arize UI中的人工审核（包括标注队列、数据集和实验）仍依赖这些配置；目前尚无针对队列的
ax
CLI命令。
说明：Arize中的人工标注会将配置定义的值附加到产品UI中的Span、数据集示例、实验相关记录和队列项上。本文档涵盖的内容：
ax annotation-configs
命令以及使用
ArizeClient.spans.update_annotations
进行批量Span更新。

Prerequisites

前置条件

Three things are needed: 
ax
CLI, an API key (env var or profile), and a space ID.
If 
ax
is not installed, not on PATH, or below version 
0.8.0
, see ax-setup.md.
macOS/Linux:
bash
ax --version && echo "--- env ---" && if [ -n "$ARIZE_API_KEY" ]; then echo "ARIZE_API_KEY: (set)"; else echo "ARIZE_API_KEY: (not set)"; fi && echo "ARIZE_SPACE_ID: ${ARIZE_SPACE_ID:-(not set)}" && echo "--- profiles ---" && ax profiles show 2>&1
Windows (PowerShell):
powershell
ax --version; Write-Host "--- env ---"; Write-Host "ARIZE_API_KEY: $(if ($env:ARIZE_API_KEY) { '(set)' } else { '(not set)' })"; Write-Host "ARIZE_SPACE_ID: $env:ARIZE_SPACE_ID"; Write-Host "--- profiles ---"; ax profiles show 2>&1
Proceed immediately if env var or profile has an API key. Only ask the user if both are missing.
  • No API key in env and no profile → AskQuestion: "Arize API key (https://app.arize.com/admin > API Keys)"
  • Space ID unknown → run 
    ax spaces list -o json
    to list all accessible spaces, or AskQuestion
需要三个要素：
ax
CLI、API密钥（环境变量或配置文件）和空间ID（Space ID）。
如果未安装
ax
、
ax
不在PATH中或版本低于
0.8.0
，请查看ax-setup.md。
macOS/Linux：
bash
ax --version && echo "--- env ---" && if [ -n "$ARIZE_API_KEY" ]; then echo "ARIZE_API_KEY: (已设置)"; else echo "ARIZE_API_KEY: (未设置)"; fi && echo "ARIZE_SPACE_ID: ${ARIZE_SPACE_ID:-(未设置)}" && echo "--- 配置文件 ---" && ax profiles show 2>&1
Windows（PowerShell）：
powershell
ax --version; Write-Host "--- env ---"; Write-Host "ARIZE_API_KEY: $(if ($env:ARIZE_API_KEY) { '(已设置)' } else { '(未设置)' })"; Write-Host "ARIZE_SPACE_ID: $env:ARIZE_SPACE_ID"; Write-Host "--- 配置文件 ---"; ax profiles show 2>&1
如果环境变量或配置文件中已有API密钥，可直接继续。仅当两者都缺失时，才需要询问用户。
  • 环境变量中无API密钥且无配置文件 → 询问问题："请提供Arize API密钥（获取地址：https://app.arize.com/admin > API Keys）"
  • 未知空间ID → 运行
    ax spaces list -o json
    列出所有可访问的空间，或询问问题

Concepts

核心概念

What is an Annotation Config?

什么是标注配置（Annotation Config）？

An annotation config defines the schema for a single type of human feedback label. Before anyone can annotate a span, dataset record, experiment output, or queue item, a config must exist for that label in the space.
FieldDescription
NameDescriptive identifier (e.g. 
Correctness
, 
Helpfulness
). Must be unique within the space.
Type
categorical
(pick from a list), 
continuous
(numeric range), or 
freeform
(free text).
ValuesFor categorical: array of 
{"label": str, "score": number}
pairs.
Min/Max ScoreFor continuous: numeric bounds.
Optimization DirectionWhether higher scores are better (
maximize
) or worse (
minimize
). Used to render trends in the UI.
标注配置定义了单一类型人工反馈标签的Schema。在任何人可以标注Span、数据集记录、实验输出或队列项之前，空间中必须存在该标签对应的配置。
字段说明
名称（Name）描述性标识符（例如
Correctness
、
Helpfulness
）。在空间内必须唯一。
类型（Type）
categorical
（从列表中选择）、
continuous
（数值范围）或
freeform
（自由文本）。
值（Values）针对分类型：
{"label": str, "score": number}
格式的数组。
分数最小值/最大值（Min/Max Score）针对连续型：数值边界。
优化方向（Optimization Direction）分数越高越好（
maximize
）还是越差（
minimize
）。用于在UI中呈现趋势。

Where labels get applied (surfaces)

标签的应用场景

SurfaceTypical path
Project spansPython SDK 
spans.update_annotations
(below) and/or the Arize UI
Dataset examplesArize UI (human labeling flows); configs must exist in the space
Experiment outputsOften reviewed alongside datasets or traces in the UI — see arize-experiment, arize-dataset
Annotation queue itemsArize UI; configs must exist — no 
ax
queue commands documented here yet
Always ensure the relevant annotation config exists in the space before expecting labels to persist.
场景典型路径
项目SpanPython SDK的
spans.update_annotations
（下文介绍）和/或Arize UI
数据集示例Arize UI（人工标注流程）；空间中必须存在对应的配置
实验输出通常在UI中与数据集或轨迹一起审核——参考arize-experiment、arize-dataset
标注队列项Arize UI；必须存在对应的配置——本文档暂未涵盖
ax
队列命令
在期望标签能够持久化之前，请务必确保空间中存在相关的标注配置。

Basic CRUD: Annotation Configs

标注配置的基础CRUD操作

List

列出配置

bash
ax annotation-configs list --space-id SPACE_ID
ax annotation-configs list --space-id SPACE_ID -o json
ax annotation-configs list --space-id SPACE_ID --limit 20
bash
ax annotation-configs list --space-id SPACE_ID
ax annotation-configs list --space-id SPACE_ID -o json
ax annotation-configs list --space-id SPACE_ID --limit 20

Create — Categorical

创建分类型配置

Categorical configs present a fixed set of labels for reviewers to choose from.
bash
ax annotation-configs create \
  --name "Correctness" \
  --space-id SPACE_ID \
  --type categorical \
  --values '[{"label": "correct", "score": 1}, {"label": "incorrect", "score": 0}]' \
  --optimization-direction maximize
Common binary label pairs:
  • correct
    / 
    incorrect
  • helpful
    / 
    unhelpful
  • safe
    / 
    unsafe
  • relevant
    / 
    irrelevant
  • pass
    / 
    fail
分类型配置为审核人员提供一组固定的标签供选择。
bash
ax annotation-configs create \
  --name "Correctness" \
  --space-id SPACE_ID \
  --type categorical \
  --values '[{"label": "correct", "score": 1}, {"label": "incorrect", "score": 0}]' \
  --optimization-direction maximize
常见的二元标签对：
  • correct
    / 
    incorrect
  • helpful
    / 
    unhelpful
  • safe
    / 
    unsafe
  • relevant
    / 
    irrelevant
  • pass
    / 
    fail

Create — Continuous

创建连续型配置

Continuous configs let reviewers enter a numeric score within a defined range.
bash
ax annotation-configs create \
  --name "Quality Score" \
  --space-id SPACE_ID \
  --type continuous \
  --minimum-score 0 \
  --maximum-score 10 \
  --optimization-direction maximize
连续型配置允许审核人员在定义的范围内输入数值分数。
bash
ax annotation-configs create \
  --name "Quality Score" \
  --space-id SPACE_ID \
  --type continuous \
  --minimum-score 0 \
  --maximum-score 10 \
  --optimization-direction maximize

Create — Freeform

创建自由文本型配置

Freeform configs collect open-ended text feedback. No additional flags needed beyond name, space, and type.
bash
ax annotation-configs create \
  --name "Reviewer Notes" \
  --space-id SPACE_ID \
  --type freeform
自由文本型配置用于收集开放式文本反馈。除名称、空间和类型外，无需其他额外参数。
bash
ax annotation-configs create \
  --name "Reviewer Notes" \
  --space-id SPACE_ID \
  --type freeform

Get

获取配置详情

bash
ax annotation-configs get ANNOTATION_CONFIG_ID
ax annotation-configs get ANNOTATION_CONFIG_ID -o json
bash
ax annotation-configs get ANNOTATION_CONFIG_ID
ax annotation-configs get ANNOTATION_CONFIG_ID -o json

Delete

删除配置

bash
ax annotation-configs delete ANNOTATION_CONFIG_ID
ax annotation-configs delete ANNOTATION_CONFIG_ID --force   # skip confirmation
Note: Deletion is irreversible. Any annotation queue associations to this config are also removed in the product (queues may remain; fix associations in the Arize UI if needed).
bash
ax annotation-configs delete ANNOTATION_CONFIG_ID
ax annotation-configs delete ANNOTATION_CONFIG_ID --force   # 跳过确认
**注意：**删除操作不可逆。产品中与此配置关联的所有标注队列关联也会被移除（队列可能保留；如有需要，请在Arize UI中修复关联）。

Applying Annotations to Spans (Python SDK)

通过Python SDK为Span添加标注

Use the Python SDK to bulk-apply annotations to project spans when you already have labels (e.g., from a review export or an external labeling tool).
python
import pandas as pd
from arize import ArizeClient

client = ArizeClient(api_key="your-api-key")
当您已有标签（例如来自审核导出或外部标注工具）时，可使用Python SDK为项目Span批量添加标注。
python
import pandas as pd
from arize import ArizeClient

client = ArizeClient(api_key="your-api-key")

Build a DataFrame with annotation columns

构建包含标注列的DataFrame

Required: context.span_id + at least one annotation.<name>.label or annotation.<name>.score

必填：context.span_id + 至少一个annotation.<name>.label或annotation.<name>.score

annotations_df = pd.DataFrame([ { "context.span_id": "span_001", "annotation.Correctness.label": "correct", "annotation.Correctness.updated_by": "reviewer@example.com", }, { "context.span_id": "span_002", "annotation.Correctness.label": "incorrect", "annotation.Correctness.updated_by": "reviewer@example.com", }, ])
response = client.spans.update_annotations( space_id="your-space-id", project_name="your-project", dataframe=annotations_df, validate=True, )

**DataFrame column schema:**

| Column | Required | Description |
|--------|----------|-------------|
| `context.span_id` | yes | The span to annotate |
| `annotation.<name>.label` | one of | Categorical or freeform label |
| `annotation.<name>.score` | one of | Numeric score |
| `annotation.<name>.updated_by` | no | Annotator identifier (email or name) |
| `annotation.<name>.updated_at` | no | Timestamp in milliseconds since epoch |
| `annotation.notes` | no | Freeform notes on the span |

**Limitation:** Annotations apply only to spans within 31 days prior to submission.

---
annotations_df = pd.DataFrame([ { "context.span_id": "span_001", "annotation.Correctness.label": "correct", "annotation.Correctness.updated_by": "reviewer@example.com", }, { "context.span_id": "span_002", "annotation.Correctness.label": "incorrect", "annotation.Correctness.updated_by": "reviewer@example.com", }, ])
response = client.spans.update_annotations( space_id="your-space-id", project_name="your-project", dataframe=annotations_df, validate=True, )

**DataFrame列Schema：**

| 列名 | 是否必填 | 说明 |
|--------|----------|-------------|
| `context.span_id` | 是 | 要标注的Span |
| `annotation.<name>.label` | 二选一 | 分类型或自由文本型标签 |
| `annotation.<name>.score` | 二选一 | 数值分数 |
| `annotation.<name>.updated_by` | 否 | 标注者标识符（邮箱或姓名） |
| `annotation.<name>.updated_at` | 否 | 时间戳（自纪元以来的毫秒数） |
| `annotation.notes` | 否 | 关于Span的自由文本备注 |

**限制：**标注仅适用于提交前31天内的Span。

---

Troubleshooting

故障排查

ProblemSolution
ax: command not found
See ax-setup.md
401 Unauthorized
API key may not have access to this space. Verify at https://app.arize.com/admin > API Keys
Annotation config not found
ax annotation-configs list --space-id SPACE_ID
409 Conflict on create
Name already exists in the space. Use a different name or get the existing config ID.
Human review / queues in UIUse the Arize app; ensure configs exist — no 
ax
annotation-queue CLI yet
Span SDK errors or missing spansConfirm 
project_name
, 
space_id
, and span IDs; use arize-trace to export spans
问题解决方案
ax: command not found
查看ax-setup.md
401 Unauthorized
API密钥可能无权访问此空间。请在https://app.arize.com/admin > API Keys验证权限
Annotation config not found
运行
ax annotation-configs list --space-id SPACE_ID
查看配置
409 Conflict on create
名称已在空间中存在。请使用其他名称或获取现有配置ID。
UI中的人工审核/队列问题使用Arize应用；确保配置存在——目前尚无
ax
标注队列CLI命令
Span SDK错误或Span缺失确认
project_name
、
space_id
和Span ID；使用arize-trace导出Span

Related Skills

相关技能

  • arize-trace: Export spans to find span IDs and time ranges
  • arize-dataset: Find dataset IDs and example IDs
  • arize-evaluator: Automated LLM-as-judge alongside human annotation
  • arize-experiment: Experiments tied to datasets and evaluation workflows
  • arize-link: Deep links to annotation configs and queues in the Arize UI
  • arize-trace：导出Span以获取Span ID和时间范围
  • arize-dataset：查找数据集ID和示例ID
  • arize-evaluator：结合人工标注的自动化LLM-as-judge功能
  • arize-experiment：与数据集和评估流程关联的实验
  • arize-link：指向Arize UI中标注配置和队列的深度链接

Save Credentials for Future Use

保存凭证供后续使用

At the end of the session, if the user manually provided any credentials during this conversation and those values were NOT already loaded from a saved profile or environment variable, offer to save them.
Skip this entirely if:
  • The API key was already loaded from an existing profile or 
    ARIZE_API_KEY
    env var
  • The space ID was already set via 
    ARIZE_SPACE_ID
    env var
How to offer: Use AskQuestion: "Would you like to save your Arize credentials so you don't have to enter them next time?" with options 
"Yes, save them"
/ 
"No thanks"
.
If the user says yes:
  1. API key — See ax-profiles.md. Run 
    ax profiles show
    to check the current state, then use 
    ax profiles create
    or 
    ax profiles update
    with the appropriate flags to save the key (and region if relevant).
  2. Space ID — See ax-profiles.md (Space ID section) to persist it as an environment variable.
在会话结束时，如果用户在此对话中手动提供了任何凭证且这些值未从已保存的配置文件或环境变量中加载，请主动提供保存选项。
完全跳过此步骤的情况：
  • API密钥已从现有配置文件或
    ARIZE_API_KEY
    环境变量加载
  • 空间ID已通过
    ARIZE_SPACE_ID
    环境变量设置
如何询问：使用询问问题："是否需要保存您的Arize凭证，以便下次无需重复输入？"，选项为
"是，保存"
/ 
"不用了，谢谢"
。
如果用户选择是：
  1. API密钥——参考ax-profiles.md。运行
    ax profiles show
    查看当前状态，然后使用
    ax profiles create
    或
    ax profiles update
    并添加适当的参数来保存密钥（如有需要，同时保存区域信息）。
  2. 空间ID——参考ax-profiles.md（空间ID章节）将其持久化为环境变量。