simulations

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Vapi Simulations

Build realistic conversation tests in five layers: a personality controls the AI tester, a scenario defines its intent and measurable outcomes, a simulation pairs them, a suite groups simulations, and a run executes them against an assistant or squad.

构建五层真实对话测试：角色设定控制AI测试者，场景定义其意图和可衡量结果，模拟任务将二者配对，测试套件对模拟任务进行分组，运行环节则针对助手或团队执行这些测试。

Source and Safety Rules

来源与安全规则

Verify live payloads against the current Vapi documentation MCP, API reference, or public OpenAPI before sending them. Simulations use the
```
/eval/simulation
```
API family.
Never print, request in chat, or embed API keys, provider secrets, credential values, private webhook URLs, or real customer data.
Treat running a simulation as an external action. It can consume credits, use concurrency, send webhooks, and call the target's real tools unless they are mocked.
Do not run, cancel, update, or delete resources unless the user clearly requests that operation. Draft configurations when mutation is not requested.
Resolve every assistant, squad, personality, scenario, simulation, suite, tool, structured-output, and credential ID from user input or the API. Never invent an ID.
Do not create legacy Test Suites. Use Evals for deterministic turn-by-turn checks and Simulations for dynamic conversations over chat or voice.

在发送实时负载前，对照当前Vapi文档MCP、API参考或公开OpenAPI进行验证。Simulations使用
```
/eval/simulation
```
API系列。
切勿打印、在聊天中请求或嵌入API密钥、提供商密钥、凭证值、私有Webhook URL或真实客户数据。
将运行模拟视为外部操作。它会消耗积分、占用并发资源、发送Webhook，并且除非工具被模拟，否则会调用目标的真实工具。
除非用户明确要求，否则请勿运行、取消、更新或删除资源。当未请求变更时，仅生成配置草稿。
从用户输入或API中解析所有助手、团队、角色设定、场景、模拟任务、测试套件、工具、结构化输出和凭证ID。切勿自行编造ID。
请勿创建旧版Test Suites。对于确定性的逐轮检查使用Evals，对于聊天或语音形式的动态对话使用Simulations。

Procedure

操作流程

Choose the test type and execution mode.
- Use Simulations for multi-turn behavior, personality variation, squad handoffs, realistic tool paths, or audio behavior.
- Use Evals instead when the requirement is an exact response, regex, fixed mock conversation, or precise tool-call argument check.
- Return a test plan or payload when the user asks to design, draft, review, or explain. Perform live mutations only when explicitly requested and
```
VAPI_API_KEY
```
  is available.
Inspect the target and existing test resources.
- Fetch the assistant or squad and identify its core paths, guardrails, tools, variables, languages, and failure behavior.
- List existing personalities, scenarios, simulations, suites, and reusable structured outputs before creating duplicates.
- Reuse an existing resource only when its intent and configuration match unambiguously. Otherwise create a clearly named new resource or ask the user to choose among plausible matches.
Design coverage before payloads.
- Start with one smoke simulation for the core path, one or two required Boolean outcomes, chat transport, and one iteration.
- Add regression simulations for repaired defects. Add separate edge cases for ambiguity, interruption, refusal, unavailable dependencies, failed tools, escalation, and handoffs.
- Keep scenario intent, personality behavior, and evaluation criteria independent so each can be reused.
- Name resources by behavior and expected outcome, not implementation details.
Define the personality.
- Prefer a suitable existing personality when available.
- When creating one, provide a complete valid assistant configuration for the AI tester. Put stable temperament, speaking style, and caller behavior in its system prompt; put the situation-specific goal in the scenario.
- Use the
```
create-assistant
```
  skill to assemble or validate the personality's assistant configuration when available.
- Configure voice and transcriber only when voice runs need them. Chat runs use the personality's model but skip its audio path.
Define the scenario and evaluations.
- Write
```
instructions
```
  as the AI tester's intent and facts. Describe the goal and constraints without scripting the target assistant's answer.
- Make each evaluation measure one observable outcome. Prefer descriptive Boolean outputs for pass/fail facts and numeric outputs for thresholds.
- Provide either
```
structuredOutputId
```
  or inline
```
structuredOutput
```
  , never both. Inline outputs require
```
name
```
  and a JSON
```
schema
```
  .
- Match the expected
```
value
```
  type to the evaluated primitive. Use
```
=
```
  or
```
!=
```
  for Boolean and string; numeric types also support
```
>
```
  ,
```
<
```
  ,
```
>=
```
  , and
```
<=
```
  .
- Keep important criteria
```
required: true
```
  . Use optional criteria only for diagnostics that must not fail the simulation.
- Object structured outputs may be evaluated through a primitive leaf using
```
path
```
  . Do not compare an object or array directly.
Isolate side effects and runtime context.
- Inspect the target's configured tools before every run. Mock any tool whose real execution could write data, contact people, spend money, or make the test non-deterministic.
- Match each
```
toolMocks[].toolName
```
  exactly. The mock
```
result
```
  is always a string; encode JSON as a string when the target expects JSON-shaped output.
- Assume every unmocked tool remains live in both chat and voice simulations.
- Put test values for
```
{{variables}}
```
  in
```
targetOverrides.variableValues
```
  . Use synthetic data and keep secrets in Vapi credentials.
- Configure
```
simulation.run.started
```
  or
```
simulation.run.ended
```
  hooks only when requested. Prefer
```
server.credentialId
```
  to inline authorization headers.
Create and verify reusable resources.
- Create in dependency order: personality and scenario, then simulation, then optional suite.
- Require
```
201
```
  for create operations. Verify returned IDs and the fields that define the test.
- For updates, fetch the current resource first. Omit unrelated scalar fields and send the complete intended value for any array being changed; suite
```
simulationIds
```
  and
```
targetAssignments
```
  replace their existing arrays.
- Re-fetch after update. Deleting a suite or other simulation resource is permanent; verify the exact ID and dependency impact first.
Run deliberately.
- Prefer
```
vapi.webchat
```
  for fast prompt, tool, and conversation-logic iteration.
- Use
```
vapi.websocket
```
  for speech recognition, voice output, interruptions, recordings, or final end-to-end validation.
- Start with one iteration. Increase iterations only to measure behavioral consistency after a single run is valid.
- Before sending the run, recap the target, simulations or suite, transport, iterations, tool mocks, and any remaining live side effects.
- Create the run with
```
POST /eval/simulation/run
```
  and require
```
201
```
  . Return the run ID and dashboard
```
url
```
  when present.
Monitor and diagnose results.
- Poll
```
GET /eval/simulation/run/{id}
```
  until
```
status
```
  is
```
ended
```
  ; do not treat
```
queued
```
  or
```
running
```
  as success.
- Fetch
```
GET /eval/simulation/run/{id}/item
```
  and inspect every item. A passing group has items to evaluate, zero failed or canceled items, and every required evaluation passes.
- Report actual versus expected values, extraction errors, skipped evaluations, failure reasons, transcript evidence, transport, and iteration number.
- Diagnose the failing layer before changing the assistant: target runtime failure, scenario ambiguity, personality behavior, tool mock mismatch, structured-output extraction, or genuine assistant behavior.
- Keep the evaluation stable when fixing the assistant. Change expected criteria only when the business requirement changed.
Handle failures honestly.

On
```
400
```
, compare the request with the current schema and correct one unambiguous validation issue before at most one retry.
On
```
401
```
or
```
403
```
, stop for authentication or permission. On
```
404
```
, report the missing dependency. On
```
409
```
or concurrency errors, inspect
```
GET /eval/simulation/concurrency
```
and active runs. On
```
5xx
```
, report the service failure.
Cancel only queued or running groups or items. Never claim a run, cancellation, mutation, or pass succeeded until the corresponding API response is verified.

选择测试类型和执行模式。
- 当需要测试多轮行为、角色变化、团队交接、真实工具路径或音频行为时，使用Simulations。
- 当需求为精确响应、正则匹配、固定模拟对话或工具调用参数精确检查时，改用Evals。
- 当用户要求设计、草拟、评审或解释时，返回测试计划或负载。仅当明确请求且
```
VAPI_API_KEY
```
  可用时，才执行实时变更操作。
检查目标对象和现有测试资源。
- 获取助手或团队信息，确定其核心路径、防护规则、工具、变量、语言和故障行为。
- 在创建重复资源前，列出已有的角色设定、场景、模拟任务、测试套件和可复用的结构化输出。
- 仅当现有资源的意图和配置完全匹配时才复用，否则创建命名清晰的新资源，或让用户在可能的匹配项中选择。
在生成负载前设计测试覆盖范围。
- 从核心路径的冒烟模拟开始，搭配1-2个必填布尔结果、聊天传输方式和1次迭代。
- 为修复的缺陷添加回归模拟。为歧义、中断、拒绝、不可用依赖、工具故障、升级和交接等情况添加单独的边缘用例。
- 保持场景意图、角色行为和评估标准相互独立，以便各自复用。
- 按行为和预期结果命名资源，而非实现细节。
定义角色设定。
- 优先选用合适的现有角色设定。
- 创建新角色时，为AI测试者提供完整有效的助手配置。将稳定的性格、说话风格和呼叫者行为放入系统提示词；将特定场景的目标放入场景设定中。
- 若可用，使用
```
create-assistant
```
  技能组装或验证角色设定的助手配置。
- 仅当需要语音运行时才配置语音和转录器。聊天运行使用角色设定的模型，但跳过音频路径。
定义场景和评估标准。
- 将
```
instructions
```
  写为AI测试者的意图和事实。描述目标和约束，但不要编写目标助手的应答脚本。
- 每个评估仅衡量一个可观察结果。优先使用描述性布尔输出表示通过/失败事实，使用数值输出表示阈值。
- 仅提供
```
structuredOutputId
```
  或内联
```
structuredOutput
```
  ，二者不可同时提供。内联输出需要
```
name
```
  和JSON
```
schema
```
  。
- 确保预期
```
value
```
  类型与评估的原始类型匹配。布尔值和字符串使用
```
=
```
  或
```
!=
```
  ；数值类型还支持
```
>
```
  、
```
<
```
  、
```
>=
```
  和
```
<=
```
  。
- 将重要标准设为
```
required: true
```
  。仅将不会导致模拟失败的诊断项设为可选标准。
- 可通过
```
path
```
  使用原始叶子节点评估结构化输出对象。请勿直接比较对象或数组。
隔离副作用和运行时上下文。
- 每次运行前检查目标的已配置工具。模拟任何真实执行会写入数据、联系人员、产生费用或导致测试非确定性的工具。
- 确保每个
```
toolMocks[].toolName
```
  完全匹配。模拟
```
result
```
  始终为字符串；当目标期望JSON格式输出时，需将JSON编码为字符串。
- 假设所有未被模拟的工具在聊天和语音模拟中均保持实时状态。
- 将
```
{{variables}}
```
  的测试值放入
```
targetOverrides.variableValues
```
  。使用合成数据，密钥保存在Vapi凭证中。
- 仅当被请求时才配置
```
simulation.run.started
```
  或
```
simulation.run.ended
```
  钩子。优先使用
```
server.credentialId
```
  而非内联授权头。
创建并验证可复用资源。
- 按依赖顺序创建：角色设定和场景，然后是模拟任务，最后是可选的测试套件。
- 创建操作要求返回
```
201
```
  状态码。验证返回的ID和定义测试的字段。
- 更新时，先获取当前资源。省略无关标量字段，对要修改的数组发送完整的预期值；测试套件的
```
simulationIds
```
  和
```
targetAssignments
```
  会替换现有数组。
- 更新后重新获取资源。删除测试套件或其他模拟资源是永久性操作；删除前需验证确切ID和依赖影响。
谨慎执行运行操作。
- 优先使用
```
vapi.webchat
```
  进行快速提示词、工具和对话逻辑迭代。
- 使用
```
vapi.websocket
```
  进行语音识别、语音输出、中断、录音或最终端到端验证。
- 从1次迭代开始。仅当单次运行有效后，才增加迭代次数以衡量行为一致性。
- 发送运行请求前，概述目标对象、模拟任务或测试套件、传输方式、迭代次数、工具模拟以及任何剩余的实时副作用。
- 使用
```
POST /eval/simulation/run
```
  创建运行请求，要求返回
```
201
```
  状态码。若存在仪表盘
```
url
```
  ，则返回运行ID和该地址。
监控并诊断结果。
- 轮询
```
GET /eval/simulation/run/{id}
```
  直到
```
status
```
  变为
```
ended
```
  ；请勿将
```
queued
```
  或
```
running
```
  视为成功状态。
- 获取
```
GET /eval/simulation/run/{id}/item
```
  并检查每个条目。通过的分组需包含待评估条目、0个失败或取消的条目，且所有必填评估均通过。
- 报告实际值与预期值、提取错误、跳过的评估、失败原因、对话记录证据、传输方式和迭代次数。
- 在修改助手前先诊断失败层级：目标运行时故障、场景歧义、角色行为、工具模拟不匹配、结构化输出提取问题，或真实的助手行为问题。
- 修复助手时保持评估稳定。仅当业务需求变更时才修改预期标准。
如实处理故障。
- 遇到
```
400
```
  错误时，将请求与当前 schema 对比，纠正一个明确的验证问题后最多重试一次。
- 遇到
```
401
```
  或
```
403
```
  错误时，停止操作并检查认证或权限。遇到
```
404
```
  错误时，报告缺失的依赖项。遇到
```
409
```
  或并发错误时，检查
```
GET /eval/simulation/concurrency
```
  和活跃运行任务。遇到
```
5xx
```
  错误时，报告服务故障。
- 仅取消排队中或运行中的分组或条目。在未验证相应API响应前，切勿声称运行、取消、变更或通过操作已成功。

API Implementation

API实现

Read Simulation API Reference before producing REST code, making a live request, configuring hooks or mocks, or interpreting run results. Use direct REST unless the current official Vapi SDK documentation explicitly exposes the required simulation resource and method; never invent SDK method names.

在生成REST代码、发送实时请求、配置钩子或模拟、解读运行结果前，请阅读Simulation API Reference。除非当前官方Vapi SDK文档明确公开了所需的模拟资源和方法，否则直接使用REST；切勿自行编造SDK方法名称。

Output Contract

输出约定

Return only the sections relevant to the request:

Test strategy: target behavior, coverage, and why Simulation rather than Eval
Resource plan: personality, scenario, evaluations, simulation, and suite
Side-effect review: mocked tools, live tools, hooks, variables, transport, iterations, and expected cost/concurrency impact
Save-ready JSON or implementation code
Created resource IDs and verified fields, when mutations succeeded
Run ID, dashboard URL, status, item counts, and per-evaluation evidence, when a run was requested
Failure diagnosis and the smallest recommended next change

仅返回与请求相关的部分：

测试策略：目标行为、覆盖范围，以及选择Simulation而非Eval的原因
资源计划：角色设定、场景、评估标准、模拟任务和测试套件
副作用审查：模拟工具、实时工具、钩子、变量、传输方式、迭代次数，以及预期成本/并发影响
可保存的JSON或实现代码
变更成功时的已创建资源ID和已验证字段
请求运行时的运行ID、仪表盘URL、状态、条目数量和各评估证据
故障诊断和最小化的推荐下一步变更

simulations

Original

Translation

Vapi Simulations

Vapi Simulations

Source and Safety Rules

来源与安全规则

Procedure

操作流程

API Implementation

API实现

Output Contract

输出约定

Public Sources

公开资源