create-agent-tests

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Agent Tests

Agent测试

Producer skill for agent tests: a coding agent executes a Markdown case file against a target environment and returns an auditable report. Verification is the executing agent's judgment plus the evidence it collects (browser, API, DB, logs, cloud, telemetry), rather than fixed deterministic assertions.

This skill authors and runs agent tests. Each case produces an auditable report — a report path, a final

PASS

FAIL

BLOCKED

ABORTED

status, and the collected evidence artifacts — for whatever invoked it to consume.

Agent测试的生成技能：由编码代理针对目标环境执行Markdown用例文件，并返回可审计的报告。验证依据是执行代理的判断及其收集的证据（浏览器、API、DB、日志、云、遥测），而非固定的确定性断言。

该技能可编写并运行Agent测试。每个用例都会生成一份可审计报告，包含报告路径、最终的

PASS

FAIL

BLOCKED

ABORTED

状态，以及收集到的证据工件，供调用方使用。

When To Use Agent Tests

何时使用Agent测试

Use agent tests when verification requires flexible, tool-driven judgment across UI, API, database, logs, files, network, or live-environment state, and a deterministic test would be premature, brittle, too expensive, or too narrow.

Especially useful for:

UI changes needing visual, interactive, browser-console, network, or trace verification.
API and DB workflows where confidence requires live requests plus persisted state inspection.
Full-stack flows crossing frontend, backend, storage, jobs, external mocks, and cleanup.
Exploratory regression checks before converting a stable path into a standard unit, contract, integration, YAML E2E, or other deterministic test.

Agent tests are not a replacement for deterministic tests. Prefer converting high-value stable agent flows into a standard deterministic test (unit, contract, integration, YAML E2E, or other) once the path stabilizes.

当验证需要跨UI、API、数据库、日志、文件、网络或实时环境状态进行灵活的、工具驱动的判断，且确定性测试尚不成熟、易失效、成本过高或覆盖范围过窄时，可使用Agent测试。

尤其适用于以下场景：

需要进行视觉、交互、浏览器控制台、网络或追踪验证的UI变更。
需要通过实时请求加持久化状态检查来确保可信度的API和DB工作流。
跨前端、后端、存储、任务、外部模拟和清理的全栈流程。
在将稳定路径转换为标准单元测试、契约测试、集成测试、YAML端到端测试或其他确定性测试之前的探索性回归检查。

Agent测试不能替代确定性测试。一旦路径稳定，建议将高价值的稳定Agent测试流程转换为标准的确定性测试（单元、契约、集成、YAML端到端等）。

Project Layout

项目结构

Agent tests live under

tests/agent/

in the target repo:

text

tests/agent/agent-test-suites.json   suite manifest (project-owned)
tests/agent/<feature>/<case>.md       case files, one logical journey each
tests/agent/run-agent-verification.ts runner/orchestrator
agent-test-reports/                    generated reports; do not hand-edit

The repo owns its real

agent-test-suites.json

, case files, fixture setup, auth/session bootstrap, CI wiring, engine secrets and MCP config, staging/production mutation policies, and cleanup ownership.

Agent测试位于目标仓库的

tests/agent/

目录下：

text

tests/agent/agent-test-suites.json   suite manifest (project-owned)
tests/agent/<feature>/<case>.md       case files, one logical journey each
tests/agent/run-agent-verification.ts runner/orchestrator
agent-test-reports/                    generated reports; do not hand-edit

仓库负责管理自身的

agent-test-suites.json

、用例文件、测试夹具设置、认证/会话引导、CI配置、引擎密钥和MCP配置、预发布/生产环境变更策略，以及清理工作的归属。

Use The Local Convention First

优先遵循本地规范

Before authoring, check for

tests/agent/agent-test-template.md

. If it exists, treat it as the authoritative local convention. Do not rediscover or replace it through broad search unless the user explicitly asks to update the convention.

If no local harness exists and an agent test is the cheapest sufficient proof, scaffold it on demand by copying the bundled starters from this skill:

<skill-dir>/assets/agent-test-template.md

→

tests/agent/agent-test-template.md

<skill-dir>/assets/agent-test-suites.example.json

→

tests/agent/agent-test-suites.example.json

<skill-dir>/assets/run-agent-verification.ts

→

tests/agent/run-agent-verification.ts

Copy these only when a project actually adopts agent tests; do not pre-seed repos that have none. Then create a real manifest from the example:

bash

cp tests/agent/agent-test-suites.example.json tests/agent/agent-test-suites.json

编写测试前，请检查是否存在

tests/agent/agent-test-template.md

。如果存在，请将其视为权威的本地规范。除非用户明确要求更新规范，否则不要通过广泛搜索重新制定或替换它。

如果没有本地测试框架，且Agent测试是成本最低且足够有效的验证方式，则可通过复制本技能中捆绑的启动文件按需搭建：

<skill-dir>/assets/agent-test-template.md

→

tests/agent/agent-test-template.md

<skill-dir>/assets/agent-test-suites.example.json

→

tests/agent/agent-test-suites.example.json

<skill-dir>/assets/run-agent-verification.ts

→

tests/agent/run-agent-verification.ts

仅当项目实际采用Agent测试时才复制这些文件；不要为尚未使用的仓库预先植入。然后根据示例创建真实的清单文件：

bash

cp tests/agent/agent-test-suites.example.json tests/agent/agent-test-suites.json

Authoring A Case

编写用例

Write each case from

<skill-dir>/assets/agent-test-template.md

(or the local template). A good case makes the executing agent unlikely to guess:

State the requirement, risk, or behavior under test and link its sources.
Fill real project context: URLs per environment, accounts/orgs, fixture and mutation policy, DB/log/cloud access, production synthetic-data policy, and cleanup ownership.
Separate environment preflight (prove the target can execute the case) from product verification (run the checks).
List concrete checks: browser pages/states, API routes and status codes, DB tables/rows, audit events, log filters, telemetry queries, timing.
Make each assertion unconditional for state the test itself creates. A check gated on an optional affordance ("if the UI offers X, verify Y") lets the agent skip Y and still PASS whenever X isn't found — silently dropping the coverage the case exists to provide. For fixtures you control (a freshly created token, org, etc.), require the affordance and assert on it directly; reserve conditional wording for genuinely environment-dependent surfaces.
Name the exact evidence each behavior requires.
Keep cases portable; bind environment specifics under the testing-environments section, not in the steps.
Never store raw secrets. Record variable names, roles, and access patterns only.

每个用例都基于

<skill-dir>/assets/agent-test-template.md

（或本地模板）编写。优质的用例应避免执行代理产生猜测：

说明测试的需求、风险或行为，并关联其来源。
填充真实的项目上下文：各环境的URL、账户/组织、测试夹具和变更策略、DB/日志/云访问权限、生产环境合成数据策略，以及清理工作的归属。
将环境预检（证明目标环境可执行用例）与产品验证（执行检查）分开。
列出具体的检查项：浏览器页面/状态、API路由和状态码、DB表/行、审计事件、日志过滤器、遥测查询、计时。
对于测试自身创建的状态，每个断言都应是无条件的。基于可选功能的检查（如“如果UI提供X，则验证Y”）会导致代理在未找到X时跳过Y仍能通过测试，从而悄悄丢失了该用例本应覆盖的范围。对于你可控的测试夹具（如新建的令牌、组织等），应要求该功能存在并直接断言；仅对真正依赖环境的场景使用条件性表述。
指明每个行为所需的具体证据。
保持用例的可移植性；将环境特定信息绑定在测试环境部分，而非步骤中。
切勿存储原始密钥。仅记录变量名、角色和访问模式。

Running

运行测试

Add a package script in the target repo, adjusted for its package manager:

json

{ "scripts": { "agent:verify": "tsx tests/agent/run-agent-verification.ts" } }

Run a suite or a single case:

bash

pnpm agent:verify --target local --suite smoke --project-name "<project name>"
pnpm agent:verify --target local --case tests/agent/<feature>/<case>.md

references/runner.md

documents the runner defaults, flags, matching environment variables, and the report status contract in full.

在目标仓库中添加一个包脚本，根据包管理器调整：

json

{ "scripts": { "agent:verify": "tsx tests/agent/run-agent-verification.ts" } }

运行测试套件或单个用例：

bash

pnpm agent:verify --target local --suite smoke --project-name "<project name>"
pnpm agent:verify --target local --case tests/agent/<feature>/<case>.md

references/runner.md

详细记录了运行器的默认值、标志、匹配的环境变量以及报告状态约定。

Report Status Contract

报告状态约定

Every case report must end with exactly one line:

text

Status: PASS
Status: FAIL
Status: BLOCKED
Status: ABORTED

```
PASS
```
/
```
FAIL
```
: environment preflight completed and product verification ran.
```
BLOCKED
```
: environment/setup could not execute the case (missing DB/log access, unreachable URL, failed login/session bootstrap, missing fixtures or approval). Do not report
```
FAIL
```
for an environment blocker.
```
ABORTED
```
: orchestration interruption only. Release gates ignore ABORTED and rerun the case; never end with PASS/FAIL/BLOCKED for an interruption.

Required cases fail the runner unless they return

PASS

When a case drives a browser, collect auditable evidence — HTML report, screenshot set, video, trace, or project-standard equivalent. Text-only browser claims are not sufficient.

每个用例报告必须以且仅以以下一行结尾：

text

Status: PASS
Status: FAIL
Status: BLOCKED
Status: ABORTED

```
PASS
```
/
```
FAIL
```
：环境预检完成，产品验证已执行。
```
BLOCKED
```
：环境/设置无法执行用例（缺少DB/日志访问权限、URL不可达、登录/会话引导失败、缺少测试夹具或审批）。对于环境阻塞情况，请勿报告
```
FAIL
```
。
```
ABORTED
```
：仅表示编排中断。发布闸门会忽略ABORTED状态并重试运行用例；中断情况下切勿以PASS/FAIL/BLOCKED结尾。

必填用例必须返回

PASS

，否则运行器判定失败。

当用例驱动浏览器时，需收集可审计的证据——HTML报告、截图集、视频、追踪记录或项目标准的等效证据。仅文本形式的浏览器验证结果是不够的。

Output

输出

A run leaves an auditable report at its report path, ending with exactly one

Status:

line, plus any collected evidence artifacts (HTML report, screenshots, video, trace, logs). That report and its evidence are the deliverable — whatever invoked this skill consumes them.

运行后会在报告路径生成一份可审计报告，以且仅以一行

Status:

结尾，外加收集到的所有证据工件（HTML报告、截图、视频、追踪记录、日志）。该报告及其证据是交付成果，供调用本技能的任何方使用。

When Not To Use

何时不使用

When a deterministic unit, contract, integration, or YAML E2E test is the cheaper, more durable proof — use the standard deterministic format instead.
When no live environment, access, or fixtures are available — record the blocker rather than authoring an unrunnable case.

当确定性单元测试、契约测试、集成测试或YAML端到端测试是成本更低、更持久的验证方式时——请改用标准的确定性测试格式。
当没有实时环境、访问权限或测试夹具可用时——记录阻塞情况，而非编写无法运行的用例。