verify-local-mcp

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Verify local MCP

验证本地MCP

Two layers, in order. The protocol layer proves the server and provider behave; the consumer layer proves an agent that has never seen the code can use the tools well. They catch different bugs — protocol testing found DataForSEO quirks (zoom-dependent empty SERPs), the consumer probe found ergonomics failures (9KB provider rows overflowing client token budgets, fractional inputs rejected upstream with raw provider errors). Do both.

分为两层，按顺序执行。协议层用于验证服务器和服务提供者的行为是否合规；消费者层用于验证从未接触过代码的代理能否顺利使用这些工具。它们能发现不同类型的问题——协议测试发现了DataForSEO的特殊情况（依赖缩放的空SERPs），消费者探针发现了易用性缺陷（9KB的服务提供者数据行超出客户端令牌预算、小数输入被上游拒绝并返回原始服务提供者错误）。请务必完成这两层测试。

1. Boot

1. 启动

.env.local

needs

AUTH_MODE=local_noauth

and

DATAFORSEO_API_KEY

(base64 of

login:password

). Never print the key.

Start
```
pnpm dev:agents
```
in the background. The server URL is branch-prefixed:
```
http://<branch-suffix>.open-seo.localhost:1355
```
(the exact URL is printed on boot; logs tee to
```
.logs/dev-server.log
```
).
With
```
local_noauth
```
,
```
/mcp
```
needs no token. Vite hot-reloads server code, so fix → re-call without restarting.

```
.env.local
```
文件需要配置
```
AUTH_MODE=local_noauth
```
和
```
DATAFORSEO_API_KEY
```
（格式为
```
login:password
```
的base64编码）。切勿泄露该密钥。
在后台启动
```
pnpm dev:agents
```
。服务器URL带有分支前缀：
```
http://<branch-suffix>.open-seo.localhost:1355
```
（启动时会打印确切URL；日志会输出到
```
.logs/dev-server.log
```
）。
启用
```
local_noauth
```
模式后，访问
```
/mcp
```
无需令牌。Vite会热重载服务器代码，因此修复问题后无需重启即可重新调用接口。

2. Protocol smoke (cheap, deterministic)

2. 协议冒烟测试（低成本、确定性）

Raw JSON-RPC against

/mcp

— the layer for asserting exact shapes and driving edge cases (resume taskIds, empty results, invalid inputs):

bash

curl -sS http://<url>/mcp \
  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

针对

/mcp

发起原始JSON-RPC请求——该层用于验证数据的精确格式并测试边缘场景（恢复taskId、空结果、无效输入）：

bash

curl -sS http://<url>/mcp \
  -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

tools/call: {"method":"tools/call","params":{"name":"<tool>","arguments":{...}}}


- Bootstrap: `list_projects`, then `create_project` if empty — most tools need a `projectId`.
- Test the happy path AND at least one edge per changed tool: an empty result (obscure query), an invalid identifier, and for queued tools the full lifecycle including resuming with the returned `taskId`.
- These are real, billed DataForSEO calls (metering itself short-circuits in `local_noauth` — billing needs unit tests, not this). Keep depths 10–20.


- 初始化流程：调用 `list_projects`，如果没有项目则调用 `create_project`——大多数工具都需要 `projectId`。
- 测试正常路径以及每个修改工具的至少一个边缘场景：空结果（模糊查询）、无效标识符，对于需要排队的工具，测试完整生命周期包括使用返回的 `taskId` 恢复任务。
- 这些是真实的、会产生费用的DataForSEO调用（`local_noauth` 模式下会跳过计量计费——计费逻辑需要单元测试，而非此流程）。请将深度保持在10–20之间。

3. Consumer probe (the ergonomics test)

3. 消费者探针测试（易用性测试）

Spawn a headless Claude subprocess connected as a real MCP client. Write a config:

json

{
  "mcpServers": {
    "openseo-local": { "type": "http", "url": "http://<url>/mcp" }
  }
}

Then run a NATURAL task — never name the tools; whether the model finds them from descriptions alone is the test:

bash

claude -p "<natural task a customer would ask>. Keep spend minimal: depths 10-20, one 3x3 grid max, ~10 paid calls.
Deliver two sections: 1. FINDINGS — the task result. 2. MCP FEEDBACK — critique the MCP as a first-time consumer:
were descriptions enough to pick tools without trial and error? confusing schemas, surprising output shapes or sizes,
unclear errors, credit-cost surprises? Did async/taskId flows behave as described? List anything that made you hesitate or retry." \
  --mcp-config mcp-local.json --strict-mcp-config \
  --allowedTools "mcp__openseo-local,mcp__openseo-local__*" \
  --model sonnet --max-turns 30

Use

--model sonnet

as the typical-client proxy — if sonnet navigates it cold, weaker clients likely can too. Read FINDINGS for correctness (did it get real, sensible data?) and MCP FEEDBACK for the rubric below.

启动一个作为真实MCP客户端连接的无头Claude子进程。编写配置文件：

json

{
  "mcpServers": {
    "openseo-local": { "type": "http", "url": "http://<url>/mcp" }
  }
}

然后执行一个自然语言任务——切勿指定工具名称；测试的核心是模型能否仅通过描述找到合适的工具：

bash

claude -p "<客户可能提出的自然语言任务>。尽量降低成本：深度10-20，最多一个3x3网格，约10次付费调用。
请提供两部分内容：1. 调查结果——任务执行结果。2. MCP反馈——作为首次使用者对MCP的评价：
仅通过描述能否无需反复尝试就选对工具？是否存在混淆的Schema、意外的输出格式或大小、不清晰的错误信息、超出预期的费用？异步/taskId流程是否与描述一致？列出所有导致犹豫或重试的问题。" \
  --mcp-config mcp-local.json --strict-mcp-config \
  --allowedTools "mcp__openseo-local,mcp__openseo-local__*" \
  --model sonnet --max-turns 30

使用

--model sonnet

作为典型客户端的代理——如果Sonnet能直接顺利完成操作，那么性能较弱的客户端大概率也能做到。查看“调查结果”确认正确性（是否获取到真实、合理的数据？），并根据以下评估标准查看“MCP反馈”。

4. Ergonomics rubric — what feedback to act on

4. 易用性评估标准——需要处理的反馈类型

Tool selection: the probe should pick the right tool first try. Retries or wrong-tool detours mean a description needs a sharper "use this when / not this" sentence.
Schemas: every constraint the provider enforces silently must be in the field's
```
.describe()
```
(units, whole-number requirements, defaults, what's ignored when). If the probe guessed-and-retried an input, encode the rule server-side (coerce/round) or document it — prefer coercing.
Output size: budget roughly a few KB per row. Provider rows carrying
```
popular_times
```
/attribute trees/photo URLs must be trimmed to the fields the tool's job needs; point to the single-entity tool for the full shape.
Errors: actionable, never a raw upstream field name without a hint at the fix. Failures after a billed step must keep the recovery handle (e.g. the taskId) in the message.
Async copy: descriptions must match typical latency ("usually completes within this call") and the resume path must actually work when driven by the probe, not just by curl.
Credit honesty: each description's credit sentence matches reality, including cache-hit and resume paths.

工具选择：探针应首次尝试就选对工具。如果出现重试或选错工具的情况，说明工具描述需要更明确的“适用场景/不适用场景”说明。
Schema：服务提供者静默执行的所有约束都必须在字段的
```
.describe()
```
中说明（单位、整数要求、默认值、忽略规则等）。如果探针需要反复猜测输入，那么要么在服务器端编码规则（强制转换/取整），要么明确记录规则——优先选择强制转换。
输出大小：每行数据预算约为几KB。包含
```
popular_times
```
/属性树/图片URL的服务提供者数据行必须裁剪为工具所需的字段；如需完整数据结构，可指向单实体工具。
错误信息：必须具有可操作性，绝不能仅返回原始上游字段名称而不提供修复提示。计费步骤失败后，必须在消息中保留恢复句柄（例如taskId）。
异步说明：描述必须匹配典型延迟（“通常在本次调用内完成”），且探针驱动的恢复流程必须能正常工作，而不仅仅是通过curl能工作。
费用透明度：每个描述中的费用说明必须与实际情况一致，包括缓存命中和恢复流程的费用。

5. Iterate and clean up

5. 迭代与清理

Fix findings → hot-reload picks them up → re-verify just the changed behavior via curl (cheap) → rerun the full consumer probe once per iteration round (it re-tests selection and flow, not just the fix). When done: stop the dev server background task, run the repo's tests/

ci:check

, and fold genuine provider quirks into code comments or tests so the next agent doesn't rediscover them.

修复问题→热重载会自动应用修改→通过curl重新验证仅修改的行为（低成本）→每次迭代后重新运行完整的消费者探针测试（它会重新测试工具选择和流程，而不仅仅是修复的问题）。完成后：停止后台的开发服务器任务，运行仓库的

tests/ci:check

，并将真实的服务提供者特殊情况整理到代码注释或测试中，避免后续代理重复发现这些问题。