experiments

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Run Experiments for Your Agent

为你的Agent运行实验

Experiments are pre-deployment batch tests. They run an application over a dataset and compare outputs with reusable evaluators. They are appropriate for prompt and model comparisons, regression tests, benchmarks, and CI quality gates.

实验是部署前的批量测试。它会在数据集上运行应用程序，并通过可复用的评估器对比输出结果。适用于提示词与模型对比、回归测试、基准测试以及CI质量关卡场景。

Hand Off Production Evaluation Requests

移交生产环境评估请求

If the user wants to score live traces or threads, monitor production quality, or block unsafe traffic, this is the wrong workflow.

If the
```
online-evaluations
```
skill is available, load it and follow it now.

Otherwise, tell the user to install it with:

bash

npx skills@1.5.19 add langwatch/skills/online-evaluations

Do not configure a monitor or guardrail from this skill.

如果用户希望对实时追踪记录或对话线程打分、监控生产环境质量或拦截不安全流量，此工作流并不适用。

若
```
online-evaluations
```
技能可用，请加载该技能并遵循其指引操作。
若不可用，请告知用户通过以下命令安装：
bash
```
npx skills@1.5.19 add langwatch/skills/online-evaluations
```

请勿通过本技能配置监控或防护机制。

Experiments and Scenarios

实验与场景

Use experiments for many single input and output examples with measurable results. Use the

scenarios

skill for end-to-end, multi-turn behavior and tool-calling sequences.

实验适用于大量带有可衡量结果的单输入输出示例。

scenarios

技能则适用于端到端、多轮交互行为及工具调用序列场景。

Determine Scope

确定范围

For a general request such as "test my agent":

Read the agent code, system prompt, tools, and relevant git history.
Identify the behavior most likely to regress.
Create a domain-specific dataset.
Select evaluators that measure the intended behavior.
Create and run a real experiment.
Interpret the results and recommend concrete improvements.

For a targeted request, focus on that behavior and still run the resulting experiment.

对于诸如“测试我的Agent”这类通用请求：

读取Agent代码、系统提示词、工具及相关Git历史记录。
识别最可能出现回归问题的行为。
创建特定领域的数据集。
选择能够衡量目标行为的评估器。
创建并运行真实实验。
解读结果并提出具体的改进建议。

对于针对性请求，聚焦于目标行为并运行相应实验即可。

Plan Limits

计划限制

LangWatch's free plan has limits on prompts, scenarios, evaluators, experiments, and datasets. When you hit a limit, the API returns

"Free plan limit of N reached..."

with an upgrade link.

How to handle:

Work within the limits. If 3 resources of the relevant type are allowed, create 3 meaningful ones, not 10.
Make every creation count: each one should demonstrate clear value.
Show what works FIRST. If you hit a limit, summarize what was accomplished and note that upgrading the plan raises it — point to the subscription settings on the platform (license settings instead, if
```
LANGWATCH_ENDPOINT
```
is set — self-hosted).
Do NOT delete existing resources to make room or repurpose an existing resource to evade the limit.

LangWatch的免费计划在提示词、场景、评估器、实验及数据集方面存在限制。当达到限制时，API会返回

"Free plan limit of N reached..."

并附带升级链接。

处理方式：

在限制范围内操作。若允许创建3个相关类型的资源，请创建3个有实际意义的资源，而非10个。
确保每个创建的资源都能体现明确价值。
先展示可行的内容。若达到限制，总结已完成的工作，并说明升级计划可提升限制——指向平台的订阅设置（若设置了
```
LANGWATCH_ENDPOINT
```
，则为自部署环境，指向许可证设置）。
请勿删除现有资源腾出空间，或复用现有资源规避限制。

Prerequisites

前置条件

Use

langwatch docs <path>

to read documentation as Markdown. Some useful entry points:

bash

langwatch docs                                    # Docs index
langwatch docs integration/python/guide           # Python integration
langwatch docs integration/typescript/guide       # TypeScript integration
langwatch docs prompt-management/cli              # Prompts CLI
langwatch scenario-docs                           # Scenario docs index

Discover commands with

langwatch --help

and

langwatch <subcommand> --help

. List and get commands accept

--format json

for machine-readable output. Read the docs first instead of guessing SDK APIs or CLI flags.

If no shell is available, fetch the same Markdown over plain HTTP. Append

.md

to any docs path (e.g. https://langwatch.ai/docs/integration/python/guide.md). Index: https://langwatch.ai/docs/llms.txt. Scenario index: https://langwatch.ai/scenario/llms.txt

If anything fails or confuses you while following this skill (broken commands, docs that do not match reality, errors you had to work around), ask the user for permission and run

npx langwatch report --user-approved

with a

--title

and

--summary

(or

--session <transcript.jsonl>

) to send it to the LangWatch team. No login needed, secrets and personal data are redacted locally, and it directly shapes what gets fixed.

npx langwatch report --help

explains the options.

Projects and API keys: target a real project, not a personal one.

LangWatch has two kinds of project:

Team / shared projects: real projects inside an organization. Evaluations, experiments, prompts, datasets, simulations and instrumentation must always target one of these.
Personal projects: a private "My Workspace" scratch space tied to a single user. Never send a user's evaluations, experiments or production traces here: it is for personal exploration only and is easily confused with a real project.

And two ways to authenticate:

A project API key in
.env
(
```
LANGWATCH_API_KEY
```
): the credential everything in these skills uses. It is scoped to one real project. This is the default; prefer it unless the user explicitly asks for something else.
langwatch login --device
(AI-tools / SSO): a personal device session for wrapping coding assistants (
```
langwatch claude
```
,
```
langwatch codex
```
, …). It is NOT for evaluations, prompts, datasets, scenarios or SDK instrumentation, and it points at a personal workspace. Do not run it to set up the work in these skills.

So for anything in these skills: make sure

LANGWATCH_API_KEY

for a real, shared project is in the project's

.env

— most environments already have this provisioned. Do NOT run

langwatch login

to pick a project, and never default to a personal project. If

LANGWATCH_ENDPOINT

is set, they are self-hosted, use that endpoint instead of app.langwatch.ai.

Read the experiment documentation before writing code:

bash

langwatch docs evaluations/experiments/overview
langwatch docs evaluations/experiments/sdk

使用

langwatch docs <path>

命令以Markdown格式阅读文档。以下是一些实用的入口：

bash

langwatch docs                                    # 文档索引
langwatch docs integration/python/guide           # Python集成指南
langwatch docs integration/typescript/guide       # TypeScript集成指南
langwatch docs prompt-management/cli              # 提示词管理CLI
langwatch scenario-docs                           # 场景文档索引

通过

langwatch --help

和

langwatch <subcommand> --help

查看命令列表。列表和获取类命令支持

--format json

参数以输出机器可读格式。请先阅读文档，而非猜测SDK API或CLI参数。

若无法使用Shell，可通过普通HTTP获取相同的Markdown文档。在任意文档路径后添加

.md

即可（例如：https://langwatch.ai/docs/integration/python/guide.md）。文档索引：https://langwatch.ai/docs/llms.txt。场景索引：https://langwatch.ai/scenario/llms.txt

若在遵循本技能操作过程中遇到任何问题或困惑（命令失效、文档与实际不符、需要解决的错误），请先征得用户许可，然后运行

npx langwatch report --user-approved

命令，并附带

--title

和

--summary

参数（或

--session <transcript.jsonl>

），将问题反馈给LangWatch团队。无需登录，机密信息和个人数据会在本地脱敏处理，反馈内容将直接用于问题修复。

npx langwatch report --help

可查看详细选项。

项目与API密钥：请针对真实项目，而非个人项目。

LangWatch有两种类型的项目：

团队/共享项目：组织内的真实项目。评估、实验、提示词、数据集、模拟及工具必须始终指向此类项目。
个人项目：绑定单个用户的私有“我的工作区”临时空间。请勿将用户的评估、实验或生产环境追踪记录发送至此：该空间仅用于个人探索，易与真实项目混淆。

两种认证方式：

.env
文件中的项目API密钥（
```
LANGWATCH_API_KEY
```
）：本技能所有操作使用的凭证。仅作用于单个真实项目。这是默认方式，除非用户明确要求，否则优先使用此方式。
langwatch login --device
（AI工具/SSO）：用于包装代码助手（
```
langwatch claude
```
、
```
langwatch codex
```
等）的个人设备会话。不适用于评估、提示词、数据集、场景或SDK工具，且指向个人工作区。请勿通过此命令设置本技能中的工作环境。

因此，对于本技能中的所有操作：确保真实共享项目的

LANGWATCH_API_KEY

已添加至项目的

.env

文件中——大多数环境已预先配置此项。请勿运行

langwatch login

命令选择项目，且切勿默认使用个人项目。若设置了

LANGWATCH_ENDPOINT

，则为自部署环境，请使用该端点而非app.langwatch.ai。

编写代码前，请阅读实验相关文档：

bash

langwatch docs evaluations/experiments/overview
langwatch docs evaluations/experiments/sdk

Build a Domain-Specific Dataset

创建特定领域的数据集

The examples must match what the application actually does. Read the system prompt, function signatures, tools, and knowledge sources first.

Good examples resemble real requests to this application and cover normal cases, edge cases, and past failures. Never use generic trivia such as "What is 2+2?" or "What is the capital of France?" unless the application itself is a trivia system.

If an existing LangWatch dataset is appropriate, inspect it with

langwatch dataset list --format json

and

langwatch dataset get --help

. Otherwise create the dataset in code or use the

datasets

skill.

示例必须与应用程序的实际功能匹配。请先读取系统提示词、函数签名、工具及知识来源。

优质示例应与该应用程序的真实请求相似，涵盖正常场景、边缘场景及过往失败案例。除非应用程序本身是问答系统，否则切勿使用诸如“2+2等于多少？”或“法国的首都是哪里？”这类通用常识问题。

若已有合适的LangWatch数据集，可通过

langwatch dataset list --format json

和

langwatch dataset get --help

命令查看详情。否则，请通过代码创建数据集或使用

datasets

技能。

Create the Experiment

创建实验

Use the SDK that matches the codebase. Keep credentials in environment variables and use the project's existing dependency manager.

使用与代码库匹配的SDK。将凭证保存在环境变量中，并使用项目现有的依赖管理器。

Python

python

import langwatch
import pandas as pd

dataset = pd.DataFrame([
    {
        "input": "A realistic request for this application",
        "expected_output": "The expected behavior",
    },
])

experiment = langwatch.experiment.init("agent-regression")

for index, row in experiment.loop(dataset.iterrows()):
    response = my_agent(row["input"])
    experiment.evaluate(
        "ragas/response_relevancy",
        index=index,
        data={"input": row["input"], "output": response},
        settings={"model": "openai/gpt-5-mini", "max_tokens": 2048},
    )

python

import langwatch
import pandas as pd

dataset = pd.DataFrame([
    {
        "input": "针对该应用程序的真实请求示例",
        "expected_output": "预期行为结果",
    },
])

experiment = langwatch.experiment.init("agent-regression")

for index, row in experiment.loop(dataset.iterrows()):
    response = my_agent(row["input"])
    experiment.evaluate(
        "ragas/response_relevancy",
        index=index,
        data={"input": row["input"], "output": response},
        settings={"model": "openai/gpt-5-mini", "max_tokens": 2048},
    )

TypeScript

typescript

import { LangWatch } from "langwatch";

const langwatch = new LangWatch();
const dataset = [
  {
    input: "A realistic request for this application",
    expectedOutput: "The expected behavior",
  },
];

const experiment = await langwatch.experiments.init("agent-regression");

await experiment.run(dataset, async ({ item, index }) => {
  const response = await myAgent(item.input);
  await experiment.evaluate("ragas/response_relevancy", {
    index,
    data: { input: item.input, output: response },
    settings: { model: "openai/gpt-5-mini", max_tokens: 2048 },
  });
});

Read

langwatch docs evaluations/evaluators/list

before choosing an evaluator, and take the type slug from

langwatch evaluator types --format json

— never from memory. If an evaluation fails with a

validation_error

naming the slug and an

expected

list, correct it from that list and retry once. Reuse project evaluators when appropriate. A scoring function is part of the experiment, not the experiment itself.

typescript

import { LangWatch } from "langwatch";

const langwatch = new LangWatch();
const dataset = [
  {
    input: "针对该应用程序的真实请求示例",
    expectedOutput: "预期行为结果",
  },
];

const experiment = await langwatch.experiments.init("agent-regression");

await experiment.run(dataset, async ({ item, index }) => {
  const response = await myAgent(item.input);
  await experiment.evaluate("ragas/response_relevancy", {
    index,
    data: { input: item.input, output: response },
    settings: { model: "openai/gpt-5-mini", max_tokens: 2048 },
  });
});

选择评估器前，请阅读

langwatch docs evaluations/evaluators/list

，并从

langwatch evaluator types --format json

命令的输出中获取类型标识——切勿凭记忆选择。若评估因

validation_error

失败并指明标识及

expected

列表，请根据该列表修正后重试一次。适当时可复用项目中的评估器。评分函数是实验的一部分，而非实验本身。

Run and Verify

运行与验证

Always execute the experiment. An unrun experiment is incomplete.

Python script: run it with the project's Python environment.
Notebook: execute all cells, for example with
```
jupyter nbconvert --to notebook --execute
```
.
TypeScript: run it with the project's package manager, for example
```
pnpm exec tsx experiment.ts
```
.

After it runs, verify the result with the CLI:

bash

langwatch experiment list --format json

If the CLI supports a more specific read or run for the installed version, discover it with

langwatch experiment --help

before using it.

务必执行实验。未运行的实验是不完整的。

Python脚本：使用项目的Python环境运行。
Notebook：执行所有单元格，例如使用
```
jupyter nbconvert --to notebook --execute
```
命令。
TypeScript：使用项目的包管理器运行，例如
```
pnpm exec tsx experiment.ts
```
。

运行完成后，通过CLI验证结果：

bash

langwatch experiment list --format json

若已安装版本的CLI支持更具体的读取或运行命令，请先通过

langwatch experiment --help

查看详情后再使用。

Consultant Mode

顾问模式

After delivering initial results, transition to consultant mode to help the user get maximum value.

Phase 1: read first. Before generating ANY content: read the codebase end-to-end (every system prompt, function, tool definition), study git history for agent-related changes (

git log --oneline -30

, then drill into prompt/agent/eval-related commits because the WHY in commit messages matters more than the WHAT), and read READMEs and comments for domain context.

Phase 2: quick wins. Generate best-effort content based on what you learned. Run everything, iterate until green. Show the user what works and create the a-ha moment.

Phase 3: go deeper. Once Phase 2 lands, summarize what you delivered, then suggest 2-3 specific improvements grounded in the codebase: domain edge cases, areas that need expert terminology or real data, integration points (APIs, databases, file uploads), or regression patterns from git history that deserve test coverage. Ask light questions with options, not open-ended ("Want scenarios for X or Y?", "I noticed Z was a recurring issue. Add a regression test?", "Do you have real customer queries I could use?"). Respect "that's enough" and wrap up cleanly.

Do NOT ask permission before Phase 1 and 2. Deliver value first. Do NOT ask generic questions or overwhelm with too many suggestions. Do NOT generate generic datasets. Everything must reflect the actual domain.

交付初始结果后，切换至顾问模式以帮助用户获取最大价值。

阶段1：先阅读。在生成任何内容之前：完整阅读代码库（所有系统提示词、函数、工具定义），研究与Agent相关的Git历史记录（

git log --oneline -30

，然后深入查看与提示词/Agent/评估相关的提交，因为提交信息中的原因比内容更重要），并阅读README和注释以了解领域背景。

阶段2：快速见效。基于所学内容生成最佳内容。运行所有内容，迭代直至成功。向用户展示可行的成果，创造顿悟时刻。

阶段3：深入挖掘。阶段2完成后，总结已交付的内容，然后基于代码库提出2-3个具体的改进建议：领域边缘场景、需要专业术语或真实数据的领域、集成点（API、数据库、文件上传），或Git历史记录中值得测试覆盖的回归模式。提出带有选项的轻量问题，而非开放式问题（例如：“需要为X或Y创建场景吗？”、“我注意到Z是一个反复出现的问题，是否需要添加回归测试？”、“你有真实的客户查询示例可以使用吗？”）。尊重用户“足够了”的反馈，干净利落地结束工作。

在阶段1和阶段2之前无需征得许可。先交付价值。请勿提出通用问题或过多建议。请勿生成通用数据集。所有内容必须反映实际领域。

Common Mistakes

常见错误

Do not configure production monitoring or guardrails from this skill.
Do not call a batch run an online evaluation.
Do not use placeholder datasets.
Do not guess SDK APIs when the installed documentation is available.
Do not stop after writing the experiment. Run it and inspect the real result.

请勿通过本技能配置生产环境监控或防护机制。
请勿将批量运行称为在线评估。
请勿使用占位符数据集。
请勿在已有安装文档的情况下猜测SDK API。
编写完实验后请勿停止，务必运行并检查真实结果。