online-evaluations

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Set Up Online Evaluations and Guardrails

配置在线评估与防护护栏

Online evaluations apply reusable evaluators to production traffic:

An online evaluation measures live traces or threads asynchronously.
A guardrail runs synchronously and can stop or replace unsafe traffic.

在线评估会将可复用的评估器应用于生产流量：

在线评估会异步测量实时追踪记录或对话线程。
防护护栏（guardrail）会同步运行，可阻止或替换不安全流量。

Hand Off Batch Testing Requests

转交批量测试请求

If the user wants to test a dataset, compare prompts or models, benchmark, or create a CI quality gate, this is the wrong workflow.

If the
```
experiments
```
skill is available, load it and follow it now.

Otherwise, tell the user to install it with:

bash

npx skills@1.5.19 add langwatch/skills/experiments

Do not create a batch experiment from this skill.

如果用户想要测试数据集、比较提示词或模型、进行基准测试，或创建CI质量门控，那么本工作流并不适用。

如果
```
experiments
```
技能可用，请加载该技能并遵循其指引操作。
若不可用，请告知用户通过以下命令安装：
bash
```
npx skills@1.5.19 add langwatch/skills/experiments
```

请勿通过本技能创建批量实验。

Choose the Production Workflow

选择生产环境工作流

Use an online evaluation when the user wants continuous scoring, quality trends, sampling, or evaluation by trace or thread.

Use a guardrail when the result must affect the request or response immediately, such as jailbreak detection, PII blocking, or policy enforcement.

If the user's wording is broad, inspect the application and choose the safer non-blocking online evaluation unless they explicitly require synchronous enforcement.

当用户需要持续打分、质量趋势监控、流量采样，或按追踪记录/对话线程进行评估时，使用在线评估。

当需要立即影响请求或响应结果时（如越狱检测、PII拦截、政策执行），使用防护护栏。

如果用户表述宽泛，请检查应用情况，优先选择更安全的非阻塞式在线评估，除非用户明确要求同步执行管控。

Plan Limits

计划限制

LangWatch's free plan has limits on prompts, scenarios, evaluators, experiments, and datasets. When you hit a limit, the API returns

"Free plan limit of N reached..."

with an upgrade link.

How to handle:

Work within the limits. If 3 resources of the relevant type are allowed, create 3 meaningful ones, not 10.
Make every creation count: each one should demonstrate clear value.
Show what works FIRST. If you hit a limit, summarize what was accomplished and note that upgrading the plan raises it — point to the subscription settings on the platform (license settings instead, if
```
LANGWATCH_ENDPOINT
```
is set — self-hosted).
Do NOT delete existing resources to make room or repurpose an existing resource to evade the limit.

LangWatch免费版在提示词、场景、评估器、实验和数据集方面存在限制。当达到限制时，API会返回

"Free plan limit of N reached..."

并附带升级链接。

处理方式：

在限制范围内操作。如果允许创建3个相关类型的资源，请创建3个有实际意义的资源，而非10个。
确保每个创建的资源都能体现明确价值。
先展示可行方案。如果达到限制，总结已完成的工作，并说明升级套餐可提升限制——指向平台的订阅设置（如果设置了
```
LANGWATCH_ENDPOINT
```
，则为许可证设置，即自托管版本）。
请勿删除现有资源腾出空间，或复用现有资源规避限制。

Prerequisites

前置条件

Use

langwatch docs <path>

to read documentation as Markdown. Some useful entry points:

bash

langwatch docs                                    # Docs index
langwatch docs integration/python/guide           # Python integration
langwatch docs integration/typescript/guide       # TypeScript integration
langwatch docs prompt-management/cli              # Prompts CLI
langwatch scenario-docs                           # Scenario docs index

Discover commands with

langwatch --help

and

langwatch <subcommand> --help

. List and get commands accept

--format json

for machine-readable output. Read the docs first instead of guessing SDK APIs or CLI flags.

If no shell is available, fetch the same Markdown over plain HTTP. Append

.md

to any docs path (e.g. https://langwatch.ai/docs/integration/python/guide.md). Index: https://langwatch.ai/docs/llms.txt. Scenario index: https://langwatch.ai/scenario/llms.txt

If anything fails or confuses you while following this skill (broken commands, docs that do not match reality, errors you had to work around), ask the user for permission and run

npx langwatch report --user-approved

with a

--title

and

--summary

(or

--session <transcript.jsonl>

) to send it to the LangWatch team. No login needed, secrets and personal data are redacted locally, and it directly shapes what gets fixed.

npx langwatch report --help

explains the options.

Projects and API keys: target a real project, not a personal one.

LangWatch has two kinds of project:

Team / shared projects: real projects inside an organization. Evaluations, experiments, prompts, datasets, simulations and instrumentation must always target one of these.
Personal projects: a private "My Workspace" scratch space tied to a single user. Never send a user's evaluations, experiments or production traces here: it is for personal exploration only and is easily confused with a real project.

And two ways to authenticate:

A project API key in
.env
(
```
LANGWATCH_API_KEY
```
): the credential everything in these skills uses. It is scoped to one real project. This is the default; prefer it unless the user explicitly asks for something else.
langwatch login --device
(AI-tools / SSO): a personal device session for wrapping coding assistants (
```
langwatch claude
```
,
```
langwatch codex
```
, …). It is NOT for evaluations, prompts, datasets, scenarios or SDK instrumentation, and it points at a personal workspace. Do not run it to set up the work in these skills.

So for anything in these skills: make sure

LANGWATCH_API_KEY

for a real, shared project is in the project's

.env

— most environments already have this provisioned. Do NOT run

langwatch login

to pick a project, and never default to a personal project. If

LANGWATCH_ENDPOINT

is set, they are self-hosted, use that endpoint instead of app.langwatch.ai.

Read the relevant documentation before changing configuration or code:

bash

langwatch docs evaluations/online-evaluation/overview
langwatch docs evaluations/online-evaluation/setup-monitors
langwatch docs evaluations/guardrails/overview
langwatch docs evaluations/evaluators/list

使用

langwatch docs <path>

命令以Markdown格式阅读文档。以下是一些实用的入口：

bash

langwatch docs                                    # 文档索引
langwatch docs integration/python/guide           # Python集成指南
langwatch docs integration/typescript/guide       # TypeScript集成指南
langwatch docs prompt-management/cli              # 提示词管理CLI
langwatch scenario-docs                           # 场景文档索引

通过

langwatch --help

和

langwatch <subcommand> --help

查看可用命令。列表和查询命令支持

--format json

参数以获取机器可读的输出。请先阅读文档，不要猜测SDK API或CLI参数。

如果没有Shell环境，可通过普通HTTP获取相同的Markdown文档。在任意文档路径后添加

.md

即可（例如：https://langwatch.ai/docs/integration/python/guide.md）。文档索引：https://langwatch.ai/docs/llms.txt。场景索引：https://langwatch.ai/scenario/llms.txt

如果在遵循本技能操作时遇到任何问题（命令失效、文档与实际不符、需要自行解决的错误），请先征得用户许可，然后运行

npx langwatch report --user-approved

命令，附带

--title

和

--summary

参数（或

--session <transcript.jsonl>

）将问题发送给LangWatch团队。无需登录，敏感信息会在本地脱敏，反馈将直接用于问题修复。

npx langwatch report --help

可查看参数说明。

项目与API密钥：请指向真实项目，而非个人项目。

LangWatch有两种类型的项目：

团队/共享项目：组织内的真实项目。评估、实验、提示词、数据集、模拟和工具集成必须始终指向此类项目。
个人项目：绑定单个用户的私有"我的工作区"临时空间。切勿将用户的评估、实验或生产追踪记录发送到此处：它仅用于个人探索，容易与真实项目混淆。

有两种认证方式：

.env文件中的项目API密钥（
```
LANGWATCH_API_KEY
```
）：本技能所有操作使用的凭证。它仅对一个真实项目生效。这是默认方式，除非用户明确要求其他方式，否则优先使用。
langwatch login --device
（AI工具/SSO）：用于包装代码助手的个人设备会话（如
```
langwatch claude
```
、
```
langwatch codex
```
等）。它不适用于评估、提示词、数据集、场景或SDK集成，且指向个人工作区。请勿通过此命令设置本技能中的工作内容。

因此，对于本技能中的所有操作：确保真实共享项目的

LANGWATCH_API_KEY

已添加到项目的

.env

文件中——大多数环境已预先配置。请勿运行

langwatch login

命令选择项目，且切勿默认使用个人项目。如果设置了

LANGWATCH_ENDPOINT

，则为自托管版本，请使用该端点而非app.langwatch.ai。

在修改配置或代码前，请阅读相关文档：

bash

langwatch docs evaluations/online-evaluation/overview
langwatch docs evaluations/online-evaluation/setup-monitors
langwatch docs evaluations/guardrails/overview
langwatch docs evaluations/evaluators/list

Inspect the Existing Setup

检查现有配置

Use JSON output and inspect what already exists before creating duplicates:

bash

langwatch monitor list --format json
langwatch evaluator list --format json

Read recent traces only when they are needed to determine mappings, level, sampling, or realistic evaluator inputs. Do not send production data to a different project.

创建重复资源前，请使用JSON输出检查已存在的资源：

bash

langwatch monitor list --format json
langwatch evaluator list --format json

仅当需要确定映射、层级、采样或真实评估器输入时，才读取近期的追踪记录。请勿将生产数据发送到其他项目。

Create an Online Evaluation

创建在线评估

Discover the installed CLI contract first:

bash

langwatch monitor create --help

Then create the monitor with a descriptive name, a valid evaluator type or saved evaluator, and the correct level:

Use
```
trace
```
for per-interaction quality.
Use
```
thread
```
for multi-message outcomes and configure an appropriate idle timeout in the platform when needed.
Start with a conservative sample rate for expensive evaluators on high-volume traffic.
Use
```
ON_MESSAGE
```
for asynchronous online evaluation.

Take the evaluator type from the catalog, never from memory:

bash

langwatch evaluator types --format json

If a create still fails with a

validation_error

whose reason names the field and an

expected

list, correct that exact field from the list and retry once. That failure is yours to fix — do not ask the user to pick a type slug.

Do not guess evaluator parameters. Read the evaluator docs and the installed CLI help. If an LLM evaluator is used, verify that the target project has a model provider configured.

After creation, verify the saved resource:

bash

langwatch monitor list --format json
langwatch monitor get <monitor-id> --format json

The task is complete only when the created monitor appears with the intended evaluator, execution mode, level, sample rate, and enabled state.

首先了解已安装的CLI约定：

bash

langwatch monitor create --help

然后使用描述性名称、有效的评估器类型或已保存的评估器，以及正确的层级创建监控器：

使用
```
trace
```
层级评估单次交互的质量。
使用
```
thread
```
层级评估多消息结果，必要时在平台中配置合适的空闲超时时间。
对于高流量场景下的高成本评估器，初始时请设置保守的采样率。
使用
```
ON_MESSAGE
```
模式进行异步在线评估。

从目录中获取评估器类型，切勿凭记忆选择：

bash

langwatch evaluator types --format json

如果创建操作因

validation_error

失败（错误信息会指出字段和

expected

列表），请根据列表修正对应字段并重试一次。此类错误由你自行修复——请勿让用户选择类型标识。

不要猜测评估器参数。请阅读评估器文档和已安装的CLI帮助。如果使用LLM评估器，请确认目标项目已配置模型提供商。

创建完成后，验证已保存的资源：

bash

langwatch monitor list --format json
langwatch monitor get <monitor-id> --format json

仅当创建的监控器显示预期的评估器、执行模式、层级、采样率和启用状态时，任务才算完成。

Add a Guardrail

添加防护护栏

For platform-managed guardrails, create or edit the monitor with

AS_GUARDRAIL

after reading

langwatch monitor create --help

langwatch monitor update --help

For an in-code guardrail, follow the language-specific documentation. A Python integration has this general shape:

python

import langwatch

@langwatch.trace()
def my_agent(user_input):
    result = langwatch.evaluation.evaluate(
        "azure/jailbreak",
        name="Jailbreak detection",
        as_guardrail=True,
        data={"input": user_input},
    )
    if not result.passed:
        return "I cannot help with that request."

    return generate_response(user_input)

Treat the snippet as a shape, not a substitute for the installed docs. Preserve the application's existing error handling and decide explicitly what happens if the guardrail service is unavailable.

对于平台托管的防护护栏，请先阅读

langwatch monitor create --help

或

langwatch monitor update --help

，然后使用

AS_GUARDRAIL

参数创建或编辑监控器。

对于代码内防护护栏，请遵循语言特定的文档。Python集成的大致结构如下：

python

import langwatch

@langwatch.trace()
def my_agent(user_input):
    result = langwatch.evaluation.evaluate(
        "azure/jailbreak",
        name="Jailbreak detection",
        as_guardrail=True,
        data={"input": user_input},
    )
    if not result.passed:
        return "I cannot help with that request."

    return generate_response(user_input)

请将该代码片段视为参考结构，不要替代已安装的文档。保留应用现有的错误处理，并明确决定防护护栏服务不可用时的处理方式。

Verify Real Behavior

验证实际行为

For an online evaluation:

Send or reuse a representative traced interaction in the target project.
Confirm the monitor is enabled.
Confirm a real evaluation result appears in Online Evaluations analytics.

For a guardrail:

Run one allowed input and one input that should be blocked.
Verify the allowed path still works.
Verify the blocked path does not reach the protected operation.
Verify both outcomes are traced without exposing sensitive content.

对于在线评估：

在目标项目中发送或复用具有代表性的追踪交互。
确认监控器已启用。
确认真实的评估结果出现在在线评估分析页面中。

对于防护护栏：

运行一个允许的输入和一个应被拦截的输入。
验证允许的路径仍可正常工作。
验证被拦截的路径无法触达受保护的操作。
验证两种结果都已被追踪，且未暴露敏感内容。

Common Mistakes

常见错误

Do not create a batch experiment from this skill.
Do not describe a synchronous guardrail as asynchronous monitoring.
Do not enable an expensive evaluator on all traffic without considering sampling and cost.
Do not create duplicate monitors without inspecting the project first.
Do not claim success after saving configuration. Verify a real monitor or guardrail behavior.

请勿通过本技能创建批量实验。
请勿将同步防护护栏描述为异步监控。
未考虑采样和成本的情况下，请勿对所有流量启用高成本评估器。
未检查项目现有资源前，请勿创建重复的监控器。
保存配置后请勿直接宣称成功。请验证监控器或防护护栏的实际行为。