Vapi Simulations

Build realistic conversation tests in five layers: a personality controls the AI tester, a scenario defines its intent and measurable outcomes, a simulation pairs them, a suite groups simulations, and a run executes them against an assistant or squad.

Source and Safety Rules

Verify live payloads against the current Vapi documentation MCP, API reference, or public OpenAPI before sending them. Simulations use the
```
/eval/simulation
```
API family.
Never print, request in chat, or embed API keys, provider secrets, credential values, private webhook URLs, or real customer data.
Treat running a simulation as an external action. It can consume credits, use concurrency, send webhooks, and call the target's real tools unless they are mocked.
Do not run, cancel, update, or delete resources unless the user clearly requests that operation. Draft configurations when mutation is not requested.
Resolve every assistant, squad, personality, scenario, simulation, suite, tool, structured-output, and credential ID from user input or the API. Never invent an ID.
Do not create legacy Test Suites. Use Evals for deterministic turn-by-turn checks and Simulations for dynamic conversations over chat or voice.

Procedure

Choose the test type and execution mode.
- Use Simulations for multi-turn behavior, personality variation, squad handoffs, realistic tool paths, or audio behavior.
- Use Evals instead when the requirement is an exact response, regex, fixed mock conversation, or precise tool-call argument check.
- Return a test plan or payload when the user asks to design, draft, review, or explain. Perform live mutations only when explicitly requested and
```
VAPI_API_KEY
```
  is available.
Inspect the target and existing test resources.
- Fetch the assistant or squad and identify its core paths, guardrails, tools, variables, languages, and failure behavior.
- List existing personalities, scenarios, simulations, suites, and reusable structured outputs before creating duplicates.
- Reuse an existing resource only when its intent and configuration match unambiguously. Otherwise create a clearly named new resource or ask the user to choose among plausible matches.
Design coverage before payloads.
- Start with one smoke simulation for the core path, one or two required Boolean outcomes, chat transport, and one iteration.
- Add regression simulations for repaired defects. Add separate edge cases for ambiguity, interruption, refusal, unavailable dependencies, failed tools, escalation, and handoffs.
- Keep scenario intent, personality behavior, and evaluation criteria independent so each can be reused.
- Name resources by behavior and expected outcome, not implementation details.
Define the personality.
- Prefer a suitable existing personality when available.
- When creating one, provide a complete valid assistant configuration for the AI tester. Put stable temperament, speaking style, and caller behavior in its system prompt; put the situation-specific goal in the scenario.
- Use the
```
create-assistant
```
  skill to assemble or validate the personality's assistant configuration when available.
- Configure voice and transcriber only when voice runs need them. Chat runs use the personality's model but skip its audio path.
Define the scenario and evaluations.
- Write
```
instructions
```
  as the AI tester's intent and facts. Describe the goal and constraints without scripting the target assistant's answer.
- Make each evaluation measure one observable outcome. Prefer descriptive Boolean outputs for pass/fail facts and numeric outputs for thresholds.
- Provide either
```
structuredOutputId
```
  or inline
```
structuredOutput
```
  , never both. Inline outputs require
```
name
```
  and a JSON
```
schema
```
  .
- Match the expected
```
value
```
  type to the evaluated primitive. Use
```
=
```
  or
```
!=
```
  for Boolean and string; numeric types also support
```
>
```
  ,
```
<
```
  ,
```
>=
```
  , and
```
<=
```
  .
- Keep important criteria
```
required: true
```
  . Use optional criteria only for diagnostics that must not fail the simulation.
- Object structured outputs may be evaluated through a primitive leaf using
```
path
```
  . Do not compare an object or array directly.
Isolate side effects and runtime context.
- Inspect the target's configured tools before every run. Mock any tool whose real execution could write data, contact people, spend money, or make the test non-deterministic.
- Match each
```
toolMocks[].toolName
```
  exactly. The mock
```
result
```
  is always a string; encode JSON as a string when the target expects JSON-shaped output.
- Assume every unmocked tool remains live in both chat and voice simulations.
- Put test values for
```
{{variables}}
```
  in
```
targetOverrides.variableValues
```
  . Use synthetic data and keep secrets in Vapi credentials.
- Configure
```
simulation.run.started
```
  or
```
simulation.run.ended
```
  hooks only when requested. Prefer
```
server.credentialId
```
  to inline authorization headers.
Create and verify reusable resources.
- Create in dependency order: personality and scenario, then simulation, then optional suite.
- Require
```
201
```
  for create operations. Verify returned IDs and the fields that define the test.
- For updates, fetch the current resource first. Omit unrelated scalar fields and send the complete intended value for any array being changed; suite
```
simulationIds
```
  and
```
targetAssignments
```
  replace their existing arrays.
- Re-fetch after update. Deleting a suite or other simulation resource is permanent; verify the exact ID and dependency impact first.
Run deliberately.
- Prefer
```
vapi.webchat
```
  for fast prompt, tool, and conversation-logic iteration.
- Use
```
vapi.websocket
```
  for speech recognition, voice output, interruptions, recordings, or final end-to-end validation.
- Start with one iteration. Increase iterations only to measure behavioral consistency after a single run is valid.
- Before sending the run, recap the target, simulations or suite, transport, iterations, tool mocks, and any remaining live side effects.
- Create the run with
```
POST /eval/simulation/run
```
  and require
```
201
```
  . Return the run ID and dashboard
```
url
```
  when present.
Monitor and diagnose results.
- Poll
```
GET /eval/simulation/run/{id}
```
  until
```
status
```
  is
```
ended
```
  ; do not treat
```
queued
```
  or
```
running
```
  as success.
- Fetch
```
GET /eval/simulation/run/{id}/item
```
  and inspect every item. A passing group has items to evaluate, zero failed or canceled items, and every required evaluation passes.
- Report actual versus expected values, extraction errors, skipped evaluations, failure reasons, transcript evidence, transport, and iteration number.
- Diagnose the failing layer before changing the assistant: target runtime failure, scenario ambiguity, personality behavior, tool mock mismatch, structured-output extraction, or genuine assistant behavior.
- Keep the evaluation stable when fixing the assistant. Change expected criteria only when the business requirement changed.
Handle failures honestly.

On
```
400
```
, compare the request with the current schema and correct one unambiguous validation issue before at most one retry.
On
```
401
```
or
```
403
```
, stop for authentication or permission. On
```
404
```
, report the missing dependency. On
```
409
```
or concurrency errors, inspect
```
GET /eval/simulation/concurrency
```
and active runs. On
```
5xx
```
, report the service failure.
Cancel only queued or running groups or items. Never claim a run, cancellation, mutation, or pass succeeded until the corresponding API response is verified.

API Implementation

Read Simulation API Reference before producing REST code, making a live request, configuring hooks or mocks, or interpreting run results. Use direct REST unless the current official Vapi SDK documentation explicitly exposes the required simulation resource and method; never invent SDK method names.

Output Contract

Return only the sections relevant to the request:

Test strategy: target behavior, coverage, and why Simulation rather than Eval
Resource plan: personality, scenario, evaluations, simulation, and suite
Side-effect review: mocked tools, live tools, hooks, variables, transport, iterations, and expected cost/concurrency impact
Save-ready JSON or implementation code
Created resource IDs and verified fields, when mutations succeeded
Run ID, dashboard URL, status, item counts, and per-evaluation evidence, when a run was requested
Failure diagnosis and the smallest recommended next change

simulations

NPX Install

Tags

SKILL.md Content

Vapi Simulations

Source and Safety Rules

Procedure

API Implementation

Output Contract

Public Sources