test-cli-usability

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Test Your CLI's Agent Usability

测试 CLI 的 Agent 可用性

This recipe helps you write scenario tests that verify your CLI tool works well when operated by AI agents (Claude Code, Cursor, Codex, etc.). A CLI that's agent-friendly means:

All commands can run non-interactively (no stdin prompts that hang)
Output is parseable and informative
Error messages are clear enough for an agent to self-correct
Help text enables discovery (
```
--help
```
works on every subcommand)

本实践指南帮助你编写场景测试，验证 CLI 工具在被 AI Agent（Claude Code、Cursor、Codex 等）操作时能否正常工作。对 Agent 友好的 CLI 意味着：

所有命令都可以非交互式运行（不存在会阻塞的 stdin 提示）
输出可解析且信息明确
错误消息足够清晰，使 Agent 能够自行纠正
帮助文本支持功能发现（每个子命令都可以使用
```
--help
```
）

Prerequisites

前置条件

Install the Scenario SDK:

bash

npm install @langwatch/scenario vitest @ai-sdk/openai

安装 Scenario SDK：

bash

npm install @langwatch/scenario vitest @ai-sdk/openai

or: pip install langwatch-scenario pytest

undefined

undefined

Step 1: Identify Your CLI Commands

第 1 步：识别 CLI 命令

List every command your CLI supports. For each, note:

Does it require interactive input? (MUST have a non-interactive alternative)
What flags/options does it accept?
What does it output on success/failure?

列出 CLI 支持的所有命令。对于每个命令，记录：

是否需要交互式输入？（必须提供非交互式替代方案）
接受哪些 flags/options？
成功或失败时会输出什么？

Step 2: Write Scenario Tests

第 2 步：编写场景测试

For each command, write a scenario test where an AI agent discovers and uses it:

typescript

import scenario, { type AgentAdapter, AgentRole } from "@langwatch/scenario";
import { openai } from "@ai-sdk/openai";
import { describe, expect, it } from "vitest";

const myAgent: AgentAdapter = {
  role: AgentRole.AGENT,
  call: async (input) => {
    // Your Claude Code adapter here
  },
};

describe("CLI agent usability", () => {
  it("discovers and uses the command non-interactively", async () => {
    const result = await scenario.run({
      name: "CLI command discovery",
      description: "Agent discovers and uses the CLI to accomplish a task",
      agents: [
        myAgent,
        scenario.userSimulatorAgent({ model: openai("gpt-5-mini") }),
        scenario.judgeAgent({
          model: openai("gpt-5-mini"),
          criteria: [
            "Agent used the CLI command correctly",
            "Agent did not get stuck on interactive prompts",
            "Agent did not need to pipe 'yes' or use 'expect' scripting",
          ],
        }),
      ],
    });
    expect(result.success).toBe(true);
  });
});

为每个命令编写一个场景测试，让 AI Agent 发现并使用它：

typescript

import scenario, { type AgentAdapter, AgentRole } from "@langwatch/scenario";
import { openai } from "@ai-sdk/openai";
import { describe, expect, it } from "vitest";

const myAgent: AgentAdapter = {
  role: AgentRole.AGENT,
  call: async (input) => {
    // Your Claude Code adapter here
  },
};

describe("CLI agent usability", () => {
  it("discovers and uses the command non-interactively", async () => {
    const result = await scenario.run({
      name: "CLI command discovery",
      description: "Agent discovers and uses the CLI to accomplish a task",
      agents: [
        myAgent,
        scenario.userSimulatorAgent({ model: openai("gpt-5-mini") }),
        scenario.judgeAgent({
          model: openai("gpt-5-mini"),
          criteria: [
            "Agent used the CLI command correctly",
            "Agent did not get stuck on interactive prompts",
            "Agent did not need to pipe 'yes' or use 'expect' scripting",
          ],
        }),
      ],
    });
    expect(result.success).toBe(true);
  });
});

Step 3: Assert No Interactive Workarounds

第 3 步：断言未使用交互式变通方案

Add this assertion to every test:

typescript

function assertNoInteractiveWorkarounds(state) {
  const output = state.messages.map(m =>
    typeof m.content === 'string' ? m.content : JSON.stringify(m.content)
  ).join('\n');

  expect(output).not.toMatch(/echo\s+["']?[yY](?:es)?["']?\s*\|/);
  expect(output).not.toMatch(/\byes\s*\|/);
  expect(output).not.toMatch(/expect\s+-c/);
  expect(output).not.toMatch(/printf\s+["']\\n["']\s*\|/);
}

If this assertion fails, your CLI has an interactivity bug -- add

--yes

--force

, or

--non-interactive

flags to the offending commands.

将此断言添加到每个测试中：

typescript

function assertNoInteractiveWorkarounds(state) {
  const output = state.messages.map(m =>
    typeof m.content === 'string' ? m.content : JSON.stringify(m.content)
  ).join('\n');

  expect(output).not.toMatch(/echo\s+["']?[yY](?:es)?["']?\s*\|/);
  expect(output).not.toMatch(/\byes\s*\|/);
  expect(output).not.toMatch(/expect\s+-c/);
  expect(output).not.toMatch(/printf\s+["']\\n["']\s*\|/);
}

如果此断言失败，说明你的 CLI 存在交互式问题——请为有问题的命令添加

--yes

、

--force

或

--non-interactive

flag。

Step 4: Test Error Recovery

第 4 步：测试错误恢复

Write scenarios where the agent makes a mistake and must recover:

Wrong command name -> agent reads
```
--help
```
and self-corrects
Missing required argument -> agent reads error message and retries
Authentication failure -> agent follows instructions in error output

编写 Agent 犯错后必须自行恢复的场景：

命令名错误 -> Agent 读取
```
--help
```
并自行纠正
缺少必需参数 -> Agent 读取错误消息并重试
身份验证失败 -> Agent 按照错误输出中的说明操作

Common Mistakes

常见错误

Do NOT make commands that require stdin for essential operations -- always provide flag alternatives
Do NOT use interactive prompts for confirmation without a
```
--yes
```
or
```
--force
```
flag
Do NOT output errors without actionable guidance (the agent needs to know how to fix it)
DO make
```
--help
```
comprehensive on every subcommand
DO use non-zero exit codes for failures (agents check exit codes)
DO output structured information (the agent can parse it)

不要让命令的关键操作依赖 stdin——始终提供 flag 替代方案
不要在没有
```
--yes
```
或
```
--force
```
flag 的情况下使用交互式确认提示
不要输出缺少可操作指导的错误信息（Agent 需要知道如何修复）
务必确保每个子命令的
```
--help
```
内容全面
务必在失败时使用非零退出码（Agent 会检查退出码）
务必输出结构化信息（便于 Agent 解析）