testing-livekit-agents

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Testing LiveKit agents

测试 LiveKit agents

Turn-level tests are the cheapest lasting verification an agent can have. They run in the user's existing suite, in text mode, and they're fast enough for every commit. The framework's helper names and signatures change, so look them up with

reading-livekit-docs

before writing, and use the testing docs page for every API detail this skill leaves out.

Every test has the same shape: start a test session with the agent under test, run one user turn, and assert on the events that turn produced.

Turn 级测试是 agent 能拥有的最廉价的持久验证。它们运行在用户现有测试套件中，以文本模式运行，并且快到适合每次提交。框架的辅助函数名称和签名会变化，因此在编写前请用

reading-livekit-docs

查询它们，并使用测试文档页面获取本 skill 未包含的所有 API 细节。

每个测试都有相同形状：启动一个测试会话，包含被测 agent，运行一个 user turn，并对该 turn 产生的事件进行断言。

What a test asserts on

测试断言什么

A turn produces a sequence of events. A simple turn is one message. A more typical one is a tool call, its output, maybe a handoff, and then a message. You write the test by walking that sequence in order, asserting on each event, and then asserting the turn has nothing more.

There are three kinds of assertion, and most of this skill is knowing which one to use.

Structural assertions check message roles, that a tool was called, its arguments, what it returned, and that a handoff to a specific agent happened. They're deterministic and fail for exactly one reason, so prefer them.

Judged assertions use an LLM-judge helper that gives one message and an intent string to a model and asks whether they match. Use them for the content of a reply, which you can't assert exactly. Describe the intent by outcome ("tells the user the booking is confirmed and gives the time"), not by wording.

Whole-conversation assertions use a judge-group helper that runs several built-in judges concurrently over the whole chat history and aggregates their verdicts. The built-in judges cover dimensions like grounding, relevance, safety, task completion, and tool use; the docs list the current set. Use this when the question spans several turns.

Rules of thumb:

Assert structure first and judge only what's left. A judged assertion that could have been structural is slower and flakier.
Close the turn. Assert there are no further events. Otherwise a test can pass while the agent also does something you didn't intend.
One behavior per test. When a test that asserts six things fails, it tells you very little.

一个 turn 产生一系列事件。简单 turn 是一条消息。更典型的 turn 是工具调用、其输出、可能一次 handoff，然后一条消息。你通过按顺序遍历该序列、对每个事件断言、然后断言该 turn 没有更多事件来编写测试。

有三种断言，本 skill 的大部分内容在于知道该用哪一种。

结构化（Structural） 断言检查消息角色、某个工具是否被调用、其参数、返回内容，以及是否发生了到特定 agent 的 handoff。它们是确定性的，并且只因一个原因失败，因此优先使用。

评判式（Judged） 断言使用 LLM-judge 辅助函数，将一个消息和一个意图字符串交给模型，询问它们是否匹配。用于回复的内容，因为你无法精确断言。用结果来描述意图（“告诉用户预订已确认并给出时间”），而不是用措辞。

整会话（Whole-conversation） 断言使用 judge-group 辅助函数，在完整聊天历史上并发运行多个内置评判器并聚合它们的判定。内置评判器覆盖 grounding、相关性、安全性、任务完成度和工具使用等维度；文档列出了当前集合。当问题跨多个 turn 时使用。

经验法则：

先断言结构，只评判剩下的。 一个本可以是结构化的评判式断言更慢且更不稳定。
关闭 turn。 断言没有更多事件。否则测试可能通过，同时 agent 还做了你未预期的事情。
每个测试只测一个行为。 当一个断言六件事的测试失败时，它几乎不能告诉你什么问题。

Mocking tools

Mock 工具

Tests shouldn't hit real backends, since that makes a suite slow and nondeterministic. Override the tools for the agent under test and return fixed values.

Two things beyond the API:

Mock failures as well as successes. Returning an error from a mock makes the tool raise, which lets you test what the agent says when a backend is down. Most agents lack this test, and it's the behavior users notice most.
Scope matters. By default, mocks apply in a block around your own
```
run()
```
calls, which is what a test needs. A session-scoped form also exists for a session that runs on its own and needs mocks active for its whole lifetime. Simulation entrypoints use that form, tests don't. See
```
writing-livekit-scenarios
```
.

Mocking only changes execution. The model still sees the real tool schemas, so tool selection is still under test.

测试不应命中真实后端，因为那会使套件变慢且不确定。覆盖被测 agent 的工具并返回固定值。

API 之外还有两件事：

同时 mock 失败和成功。 从 mock 返回错误会使工具抛出异常，从而让你测试 agent 在后端宕机时说什么。大多数 agent 缺少这种测试，而这也是用户最注意的行为。
作用域很重要。 默认情况下，mocks 在你自己的
```
run()
```
调用周围的块内生效，这正是测试所需的。也存在一种 session-scoped 形式，用于独立运行且需要在整个生命周期内保持 mocks 生效的会话。模拟入口点使用那种形式，测试不用。参见
```
writing-livekit-scenarios
```
。

Mock 只改变执行。模型仍然看到真实的工具 schema，因此工具选择仍在测试之下。

Multi-turn and seeded history

多轮和预置历史

There are two ways to test behavior that depends on earlier turns:

Run the turns. Each turn adds to the conversation history, so the second turn sees the first. Use this when the path matters: collecting details across turns, the user changing their mind mid-flow, a handoff carrying context.
Seed the history. Build a chat history and give it to the agent to start at the state you care about. Use this when the setup turns aren't what you're testing. It's faster and less brittle than replaying five turns to reach the sixth.

有两种方式测试依赖较早 turn 的行为：

运行这些 turn。 每个 turn 都会添加到对话历史中，因此第二个 turn 能看到第一个。当路径很重要时使用：跨 turn 收集细节、用户在流程中途改变主意、携带上下文的 handoff。
预置历史（Seed the history）。 构建一个聊天历史并交给 agent，让它从你关心的状态开始。当设置 turn 不是你要测试的内容时使用这种方法。它比重放五个 turn 到达第六个更快、更不易碎。

What to test

测试什么

Roughly in order of value:

The behavior the user asked for. Every agent needs at least this.
Tool invocation: the right tool with the right arguments for a representative request.
Tool failure: what the agent says when a tool errors or returns nothing.
Refusals and limits: the agent declines what it should decline and doesn't make up data it can't have. The refusal is the pass condition.
Handoffs: the transition fires when it should, and the next agent has what it needs.
Every bug you've fixed. Without a test, a fixed bug can come back unnoticed.

大致按价值排序：

用户要求的行为。 每个 agent 至少需要这个。
工具调用：针对代表性请求调用正确的工具和正确的参数。
工具失败：当工具出错或返回空时 agent 说什么。
拒绝和限制：agent 拒绝它应当拒绝的内容，不编造它无法获得的数据。拒绝就是通过条件。
Handoffs：转接应在触发时触发，并且下一个 agent 拥有它需要的东西。
你修复的每个 bug。 没有测试，已修复的 bug 可能悄悄复发。

Three kinds of evidence, kept distinct

三种证据，保持区分

A test suite for an agent that does things — books, edits, confirms — needs three kinds of test, and confusing them is how a green suite ships a broken agent:

Deterministic application tests prove guards, no-ops, empty-vs-unknown values, corrections, storage failure, and retries, against the application core directly. Fast, exact, no model.
Real-SDK tests with a scripted provider prove event wiring and tool plumbing: that the listener fires, that the tool receives what the session sends. A scripted provider proves mechanics, never language understanding — its exact sentences are fixtures, not requirements.
Ordinary-language tests with the real model prove the model can complete the task from how callers actually talk: paraphrases absent from the prompt, corrections mixed with agreement, the same short reply answering different questions.

Shortcuts that look like tests and aren't: calling a helper or filling private state instead of driving the interaction; telling the test caller which tool to name; mocking the very mutation whose correctness is under test; comparing output only with the application's own exporter; retrying a failed turn until it passes; deleting the assertion that failed.

一个为做事的 agent（预订、编辑、确认）设计的测试套件需要三种测试，混淆它们就是让绿色套件交付有问题的 agent 的方式：

确定性应用测试 直接针对应用核心证明守卫、空操作、空值 vs 未知值、更正、存储失败和重试。快速、精确、无模型。
使用脚本化提供者的真实 SDK 测试 证明事件连接和工具管道：监听器触发，工具收到会话发送的内容。脚本化提供者证明的是机制，而不是语言理解——它的准确句子是固定装置（fixtures），不是需求。
使用真实模型的自然语言测试 证明模型能从调用者实际说话方式完成任务：提示中没有的转述、与同意混合的更正、同一个简短回复回答不同问题。

看起来像测试但实际不是的捷径：调用辅助函数或填充私有状态而不是驱动交互；告诉测试调用者该命名哪个工具；mock 正在测试的正确性所依赖的那个变更；只将输出与应用自己的导出器比较；重试失败的 turn 直到通过；删除失败的断言。

Don't write tests that punish correct behavior

不要编写惩罚正确行为的测试

The most common bad agent test asserts that the agent does something it shouldn't: states data it can't know, gives a specific medical, legal, or financial recommendation, or completes a flow that should have been blocked. If the agent can only pass by misbehaving, fix the test.

For a guardrail, the pass is that the agent refuses, escalates, or declines to make something up. Write the assertion that way.

最常见的坏 agent 测试断言 agent 做了它不该做的事：说出它不可能知道的数据、给出具体的医疗、法律或财务建议，或完成一个本应被阻止的流程。如果 agent 只能通过错误行为来通过，那就修复测试。

对于护栏，通过条件是 agent 拒绝、升级或拒绝编造。请以这种方式编写断言。

Where this sits

位置

Interactive poking while building →
```
debugging-livekit-agents
```
. Faster loop, no assertions kept.
Turn-level, deterministic, every commit → here. Cheapest lasting check.
Whole conversations graded by a simulated user, before a release →
```
running-livekit-simulations
```
. More expensive, and catches emergent behavior that turn-level tests miss.

When a simulation keeps failing the same way, the bug is usually turn-level. Write a test for it here, which pins down the cause more precisely and catches it earlier.

构建时交互式调试 →
```
debugging-livekit-agents
```
。更快的循环，不保留断言。
逐轮、确定性、每次提交 → 本 skill。最廉价的持久检查。
发布前由模拟用户评判整个对话 →
```
running-livekit-simulations
```
。更昂贵，能捕获逐轮测试遗漏的涌现行为。

当模拟持续以同样方式失败时，bug 通常是 turn 级的。在这里为它编写测试，可以更精确地定位原因并更早捕获。