momentic-maintain
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseMaintain Momentic tests
维护Momentic测试
Find the earliest point where a run entered a bad state, classify the cause from
evidence, and repair the test when a durable test change is appropriate. Use MCP
for targeted investigation, for a fresh classifier
verdict, and for automated repair and verification.
momentic ai classifymomentic ai triageDo not weaken a test to hide an application bug, missing fixture, invalid
credentials, outage, or Momentic defect. Report those causes instead of changing
the test to pass.
找到运行进入异常状态的最早节点,根据证据分类故障原因,并在需要持久化测试变更时修复测试。使用MCP进行针对性调查,使用获取全新的分类器结论,使用进行自动化修复和验证。
momentic ai classifymomentic ai triage不得通过弱化测试来掩盖应用程序bug、缺失的测试夹具、无效凭证、服务中断或Momentic缺陷。对于这些情况,应上报原因而非修改测试使其通过。
Common workflows
常见工作流
- What happened on my branch? Call with the exact
momentic_list_runsand boundedgitBranchName/start. Start without a status filter, then narrow toend,FAILED, orCANCELLED. Open exact candidates and compare with main only for a concrete cross-branch question.recovered=true - Why is main red? List runs on the actual main branch over the incident window and paginate through every page needed for the claim. Narrow with
FAILEDwhen investigating a known category, but group symptoms only after inspecting representative runs and step evidence. Read each exact run's saved classification before optionally re-classifying it.failureCategory - De-flake and reduce recovery latency. Use for quarantined tests,
momentic_quarantine_listfor successful runs that needed recovery, andrecovered=truefor retry-heavy runs (minAttemptsmeans at least one retry;2means at least two). Inspect earlier attempts, prioritize repeated same-intent failures, fix the earliest unstable postcondition, and measure again.3
Know the discovery limits:
- filters by test, branch, status, quarantine, recovery, effective failure category, minimum total attempts, and time.
momentic_list_runsuses a manual classification when present, otherwise the automated classification.failureCategory - filters by quarantine date and returns each active quarantine's latest retained failed CLI run. It does not filter by branch or classification.
momentic_quarantine_list - A category or retry threshold can still span multiple pages. Paginate through all pages needed before calling an inventory exhaustive.
pageCount
- 我的分支上发生了什么? 使用精确的和限定的
gitBranchName/start参数调用end。初始时不设置状态过滤器,之后再缩小范围到momentic_list_runs、FAILED或CANCELLED。仅在需要明确跨分支对比时,才打开候选记录并与主分支进行比较。recovered=true - 主分支为何报错? 列出事件窗口内主分支上的运行记录,遍历所有必要的分页结果。在调查已知类别时可通过
FAILED缩小范围,但需先检查代表性运行记录和步骤证据后再归纳症状。在选择重新分类前,先查看每条运行记录已保存的分类结果。failureCategory - 修复抖动问题并缩短恢复延迟 使用查看隔离测试,使用
momentic_quarantine_list查看需要恢复才能成功的运行记录,使用recovered=true查看重试频繁的运行记录(minAttempts表示至少重试1次;2表示至少重试2次)。检查早期尝试记录,优先处理重复出现的同类型故障,修复最早出现的不稳定后置条件,然后再次验证效果。3
了解查询限制:
- 可按测试、分支、状态、隔离状态、恢复状态、实际故障类别、最小尝试次数和时间进行过滤。
momentic_list_runs优先使用手动分类结果,若无则使用自动分类结果。failureCategory - 可按隔离日期过滤,返回每个有效隔离测试的最新保留失败CLI运行记录。该工具不支持按分支或分类过滤。
momentic_quarantine_list - 按类别或重试阈值过滤时仍可能涉及多页结果,需遍历所有页才能确保查询全面。
pageCount
Maintenance workflow
维护工作流
- Start from the exact run, branch, run group, commit, or local results target.
- Inspect existing automated/manual classifications and heal state first.
- Reconstruct exact failures with MCP and only the necessary run history.
- Explain the earliest divergence and broken postcondition before editing.
- Classify afresh only when useful; persist a verdict only when requested.
- Prepare the exact app and fixtures, then triage when repair is appropriate.
- Review the diff and replay evidence; re-run inconclusive repairs.
- Report diagnosis, repair, verification, delivery, and remaining blockers.
- 从指定的运行记录、分支、运行组、提交记录或本地结果目标开始。
- 先检查现有的自动/手动分类结果并修复异常状态。
- 使用MCP和必要的运行历史重现精确故障。
- 在编辑测试前,先解释最早出现的偏差和损坏的后置条件。
- 仅在有用时重新分类,仅在用户要求时保存结论。
- 准备好精确的应用程序和测试夹具,在适合修复时执行分流操作。
- 审查差异并回放证据,重新执行不确定的修复操作。
- 上报诊断结果、修复方案、验证情况、交付进度及剩余障碍。
Inspect the existing classification
检查现有分类结果
Fetch raw run metadata before MCP investigation when an API key is available:
bash
RUN_ID="<run-id>"
MOMENTIC_SERVER_URL="${MOMENTIC_SERVER:-https://api.momentic.ai}"
RUN_INSPECTION_DIR="$(mktemp -d)"
curl --fail --silent --show-error \
-H "Authorization: Bearer ${MOMENTIC_API_KEY:?Set MOMENTIC_API_KEY}" \
"${MOMENTIC_SERVER_URL}/v1/runs/${RUN_ID}/metadata" \
-o "${RUN_INSPECTION_DIR}/metadata.json"
jq '{
runId: .id,
status,
failureReason,
classification: .failureDetails.classification,
manualClassification: .failureDetails.manualClassification,
healStatus,
healDetails
}' "${RUN_INSPECTION_DIR}/metadata.json"Do not treat a missing field as proof that no human reviewed the failure until
the metadata request succeeds for the exact run and organization. Do not echo,
log, or commit the API key. Treat downloaded metadata and signed artifact URLs
as sensitive run data.
momentic_get_run当有API密钥时,在使用MCP调查前先获取原始运行元数据:
bash
RUN_ID="<run-id>"
MOMENTIC_SERVER_URL="${MOMENTIC_SERVER:-https://api.momentic.ai}"
RUN_INSPECTION_DIR="$(mktemp -d)"
curl --fail --silent --show-error \
-H "Authorization: Bearer ${MOMENTIC_API_KEY:?Set MOMENTIC_API_KEY}" \
"${MOMENTIC_SERVER_URL}/v1/runs/${RUN_ID}/metadata" \
-o "${RUN_INSPECTION_DIR}/metadata.json"
jq '{
runId: .id,
status,
failureReason,
classification: .failureDetails.classification,
manualClassification: .failureDetails.manualClassification,
healStatus,
healDetails
}' "${RUN_INSPECTION_DIR}/metadata.json"在确认指定运行记录和组织的元数据请求成功前,不要将缺失字段视为无人审核故障的证据。不要回显、记录或提交API密钥,将下载的元数据和签名工件URL视为敏感运行数据。
momentic_get_runHelpful MCP tools
实用MCP工具
momentic_get_runmomentic_list_runsgitBranchNamerecovered=truefailureCategoryminAttempts: 2momentic_quarantine_listmomentic_get_step_resultparentStepIdChainincludeTrace=truemomentic_get_test_steps_for_runmomentic_submit_result_classificationai classify --savemomentic_get_runmomentic_list_runsgitBranchNamerecovered=truefailureCategoryminAttempts: 2momentic_quarantine_listmomentic_get_step_resultparentStepIdChainincludeTrace=truemomentic_get_test_steps_for_runmomentic_submit_result_classificationai classify --saveInvestigation workflow
调查工作流
Start with the current run before relying on history.
- Call and identify the failing attempt, section (
momentic_get_run, main steps, orbeforeSteps), failing step, and anyafterSteps.parentStepIdChain - Pull the failing step result with screenshots and trace. If the step is nested, also pull the nearest parent container or module result.
- Decide whether the failing step's before-screenshot is the correct baseline for that action. If it is already wrong, walk backward through the current run until you find the step/container that produced that bad state.
- For repeated modules or repeated workflows, compare invocations inside the same current run before comparing older runs. The later failure is often caused by an earlier invocation that succeeded, recovered, or left an invalid postcondition.
- Treat successful containers with failed or recovered child steps as partial failures until you inspect the container's final after-screenshot and URL.
- Use past runs only for specific comparison questions once the current-run behavior is understood.
Before classifying, be able to answer:
- What is the test's intended behavior?
- What is the earliest divergent step/container?
- What did that step intend to do?
- Which element/state did it actually interact with or observe?
- What changed in the screenshot, URL, DOM, trace, or recovery log after the step?
- Why is the later failure a consequence of that earlier divergence?
Avoid vague root causes such as "setup was unreliable" or "the page was in the wrong state." Name the broken postcondition directly: for example, "the row-level plus button was clicked, but the app stayed on the parent page instead of opening the child-page editor; the following global assertion passed against unrelated page text, so the untargeted type step never entered the child title."
Add to先从当前运行记录入手,再依赖历史记录。
- 调用,识别失败尝试次数、失败区段(
momentic_get_run、主步骤或beforeSteps)、失败步骤及任何afterSteps。parentStepIdChain - 获取失败步骤的结果(含截图和跟踪信息)。若步骤为嵌套结构,还需获取最近的父容器或模块结果。
- 判断失败步骤的前置截图是否为该操作的正确基准状态。若已异常,则回溯当前运行记录,直到找到产生该异常状态的步骤/容器。
- 对于重复模块或重复工作流,先对比当前运行记录内的多次调用,再对比旧运行记录。后续故障通常由之前成功、恢复或留下无效后置条件的调用导致。
- 将包含失败或已恢复子步骤的成功容器视为部分故障,需检查容器的最终后置截图和URL。
- 仅在理解当前运行记录行为后,才使用历史记录进行特定对比。
分类前需明确以下问题:
- 测试的预期行为是什么?
- 最早出现偏差的步骤/容器是什么?
- 该步骤的预期操作是什么?
- 实际交互或观察到的元素/状态是什么?
- 步骤执行后,截图、URL、DOM、跟踪信息或恢复日志有何变化?
- 后续故障为何是早期偏差的结果?
避免模糊的根本原因描述,如“设置不可靠”或“页面状态错误”。直接明确损坏的后置条件,例如:“点击行级加号按钮后,应用未跳转到子页面编辑器,仍停留在父页面;后续全局‘添加到’断言匹配了无关页面文本,导致无目标的输入步骤从未进入子页面标题栏。”
Evidence standards
证据标准
- Screenshots are the default truth source for page state. Use trace fields and DOM/HTML to explain why the screenshot changed or did not change.
- Verify every causal claim. Do not say an overlay, side peek, modal, or menu was present unless the relevant before/after screenshot, URL, or DOM proves it.
- Separate "the target is missing now" from "the browser is in the state where that target should exist." A missing target is often a symptom of an earlier failed action.
- For click/type/action steps, record the intended action, actual interacted element when available, before/after URL, and whether the expected UI state appeared.
- For assertions, check whether the assertion is scoped enough to prove the intended state. A broad page-content assertion can pass for unrelated text.
- For recovery, inspect both the failed child step and the recovered container final state. Recovery can pass a retried assertion while leaving state that later steps did not expect.
- 截图是页面状态的默认可信来源。使用跟踪字段和DOM/HTML解释截图变化或未变化的原因。
- 验证所有因果主张。除非相关前后截图、URL或DOM能证明,否则不要断言存在覆盖层、侧边弹窗、模态框或菜单。
- 区分“目标元素当前缺失”和“浏览器处于目标元素应存在的状态”。目标元素缺失通常是早期操作失败的症状。
- 对于点击/输入/操作步骤,记录预期操作、实际交互元素(若可用)、前后URL及预期UI状态是否出现。
- 对于断言,检查断言范围是否足够证明预期状态。宽泛的页面内容断言可能因无关文本而通过。
- 对于恢复操作,检查失败子步骤和恢复后容器的最终状态。恢复操作可能通过重试断言,但留下后续步骤未预期的状态。
Run AI classification
执行AI分类
momentic ai classifyUse one selector per command:
bash
npx momentic ai classify --run-id "$RUN_ID" --output-format json
npx momentic ai classify --run-group-id "<run-group-id>"
npx momentic ai classify --git-commit "<commit-sha>"- Omit for a diagnostic re-classification. Add
--saveonly when the new verdict should replace the saved classification on the local archive or cloud run.--save - Use when the existing cached classifier result is suspected to be stale and the user explicitly needs a fresh analysis.
--no-cache - Use when resuming a batch without revisiting runs that already have saved classifications.
--skip-classified - Use for automation and multi-run bookkeeping.
--output-format json - Classify multiple related run IDs together when useful, but keep each result mapped to its exact run.
For one run in a real terminal, keep the classifier conversation open:
bash
npx momentic ai classify --run-id "$RUN_ID" --interactiveUse the chat to ask why it chose the category, what evidence it relied on, how
the current run differs from a prior run, or what a durable fix would require.
Interactive classification is single-run only, requires stdin to be a TTY, and
cannot be combined with JSON output.
momentic ai classify每个命令使用一个选择器:
bash
npx momentic ai classify --run-id "$RUN_ID" --output-format json
npx momentic ai classify --run-group-id "<run-group-id>"
npx momentic ai classify --git-commit "<commit-sha>"- 诊断性重新分类时省略。仅当新结论应替换本地归档或云端运行记录中的已保存分类结果时,才添加
--save。--save - 当怀疑现有缓存分类结果过时且用户明确需要全新分析时,使用。
--no-cache - 恢复批量处理时,使用避免重新处理已保存分类结果的运行记录。
--skip-classified - 自动化处理和多运行记录管理时,使用。
--output-format json - 必要时可同时分类多个相关运行ID,但需确保每个结果对应其精确运行记录。
在真实终端中处理单个运行记录时,保持分类会话开启:
bash
npx momentic ai classify --run-id "$RUN_ID" --interactive通过聊天询问分类器为何选择该类别、依赖哪些证据、当前运行记录与之前运行记录的差异,以及持久化修复需要什么条件。交互式分类仅支持单个运行记录,要求标准输入为TTY,且无法与JSON输出结合使用。
Prepare triage
准备分流操作
Treat application readiness as a hard precondition. Triage replays the test in
a real browser; it cannot repair a test against an unavailable app, missing
fixture, wrong environment, unseeded database, invalid account, or cold compiler.
Before triage:
- Read , the target test and its setup modules, repository scripts, and the CI workflow that normally runs the test. Use the same URL, environment, credentials, headers, files, services, and data fixtures.
momentic.config.yaml - Confirm , project configuration, dependencies, and the requested browser are available.
MOMENTIC_API_KEY - Inspect and preserve unrelated work. Decide the repair delivery before running the command; do not let an unknown dashboard default create a pull request or commit unexpectedly.
git status - Build the application before starting triage. Prefer a production or otherwise precompiled build and a stable server over Next.js, webpack, Vite, or another development server that compiles routes incrementally.
- Start every required service and wait for the actual readiness URL. Warm any unavoidable dev-only compilation path before triage and verify that the relevant route no longer shows a compile/loading screen or incurs a long first request.
- Run a focused smoke or the failing test once when necessary to prove the app, fixture, authentication, and browser are ready. Fix setup failures before blaming triage.
- Start with . Raise it only when the machine has enough CPU and memory for one browser and application workload per worker.
--parallel 1
If the repository exposes a stable server command, either start it separately
or let triage own it with and :
--start--wait-onbash
undefined将应用程序就绪视为硬性前提条件。分流操作会在真实浏览器中重放测试,无法针对不可用的应用程序、缺失的测试夹具、错误环境、未初始化数据库、无效账户或冷编译状态修复测试。
分流前需完成:
- 阅读、目标测试及其设置模块、仓库脚本和通常运行测试的CI工作流。使用相同的URL、环境、凭证、请求头、文件、服务和数据夹具。
momentic.config.yaml - 确认、项目配置、依赖项及请求的浏览器可用。
MOMENTIC_API_KEY - 检查并保留无关工作。运行命令前确定修复交付方式,避免因未知控制台默认设置意外创建拉取请求或提交记录。
git status - 执行分流前先构建应用程序。优先使用生产环境或预编译构建及稳定服务器,而非Next.js、webpack、Vite等会增量编译路由的开发服务器。
- 启动所有必需服务并等待就绪URL。若无法避免仅开发环境可用的编译路径,需提前预热并验证相关路由不再显示编译/加载界面或出现首次请求延迟过长的情况。
- 必要时先运行一次聚焦的冒烟测试或失败测试,证明应用程序、测试夹具、认证及浏览器已就绪。在归咎于分流操作前先修复设置故障。
- 初始使用。仅当机器有足够CPU和内存支持每个工作进程处理一个浏览器和应用程序工作负载时,再提高并行数。
--parallel 1
若仓库提供稳定的服务器启动命令,可单独启动或通过和让分流操作管理:
--start--wait-onbash
undefinedBuild first; use the repository's actual commands and readiness URL.
先构建;使用仓库实际的命令和就绪URL。
pnpm build
npx momentic ai triage --run-id "$RUN_ID"
--start "pnpm start"
--wait-on "http://localhost:3000/health"
--parallel 1
--yes
--on-heal-success nothing
--start "pnpm start"
--wait-on "http://localhost:3000/health"
--parallel 1
--yes
--on-heal-success nothing
Do not copy these example commands blindly. Discover the real build, start, and
readiness contract from the repository and CI configuration.pnpm build
npx momentic ai triage --run-id "$RUN_ID"
--start "pnpm start"
--wait-on "http://localhost:3000/health"
--parallel 1
--yes
--on-heal-success nothing
--start "pnpm start"
--wait-on "http://localhost:3000/health"
--parallel 1
--yes
--on-heal-success nothing
不要盲目复制示例命令。从仓库和CI配置中了解真实的构建、启动和就绪约定。Run AI triage
执行AI分流
momentic ai triageRun classification first when the user wants an explicit diagnosis, then pass
the same target to triage:
bash
npx momentic ai classify --run-id "$RUN_ID" --output-format json
npx momentic ai triage --run-id "$RUN_ID" \
--parallel 1 \
--yes \
--on-heal-success nothingFor a local run group, pass the results directory instead:
bash
npx momentic ai triage ./test-results \
--parallel 1 \
--yes \
--on-heal-success nothingBest practices:
- Pass multiple related run IDs after one so the bucketing agent can group shared causes. Do not launch one competing triage command per run.
--run-id - Set explicitly: use
--on-heal-successfor local review,nothingfor a portable diff, or a pull-request/commit behavior only when the user asked for that delivery.patch - Pass only after the environment and delivery choice are confirmed so dependency prompts do not stall an autonomous run.
--yes - Use to inspect bucketing without editing tests or saving triage metadata. Dry-run is not a repair attempt.
--dry-run --no-save - Use only when a cached heal is stale or failed and a new repair should be generated from scratch.
--regenerate-heal - Keep the command attached and monitor reasoning, tool calls, browser progress, CPU, and memory. Several minutes without browser progress can indicate app startup, cold compilation, resource pressure, or a missing fixture.
- Inspect every resulting diff. Do not accept deleted assertions, broadened checks, arbitrary sleeps, or longer timeouts that merely conceal the cause.
- Do not remove quarantine solely because triage produced a patch. Re-run the repaired test first.
Use interactive triage when the user wants to question the repair agent:
bash
npx momentic ai triage --run-id "$RUN_ID" \
--parallel 1 \
--yes \
--on-heal-success nothing \
--interactiveInteractive triage requires a TTY, cannot be combined with , and is
available only when the command runs the bucketing agent. Use it to ask why a
fix was accepted or rejected, what setup it observed, and what evidence remains
unresolved.
--jsonmomentic ai triage当用户需要明确诊断时,先执行分类,再将同一目标传入分流操作:
bash
npx momentic ai classify --run-id "$RUN_ID" --output-format json
npx momentic ai triage --run-id "$RUN_ID" \
--parallel 1 \
--yes \
--on-heal-success nothing对于本地运行组,传入结果目录即可:
bash
npx momentic ai triage ./test-results \
--parallel 1 \
--yes \
--on-heal-success nothing最佳实践:
- 在一个后传入多个相关运行ID,以便分组Agent按共同原因分组。不要为每个运行记录启动独立的分流命令。
--run-id - 显式设置:本地审核使用
--on-heal-success,可移植差异使用nothing,仅当用户要求时才选择拉取请求/提交行为。patch - 确认环境和交付方式后再传入,避免依赖提示中断自动化运行。
--yes - 使用检查分组情况,不编辑测试或保存分流元数据。试运行不执行修复尝试。
--dry-run --no-save - 当缓存修复过时或失败且需要从头生成新修复时,使用。
--regenerate-heal - 保持命令运行并监控推理过程、工具调用、浏览器进度、CPU和内存使用情况。数分钟无浏览器进度可能表明应用程序启动、冷编译、资源压力或缺失测试夹具。
- 检查所有生成的差异。不要接受删除断言、扩大检查范围、任意添加等待或延长超时时间等仅掩盖问题的修改。
- 不要仅因分流操作生成补丁就解除隔离,需先重新运行修复后的测试。
当用户希望向修复Agent提问时,使用交互式分流:
bash
npx momentic ai triage --run-id "$RUN_ID" \
--parallel 1 \
--yes \
--on-heal-success nothing \
--interactive交互式分流需要TTY,无法与结合使用,仅当命令运行分组Agent时可用。可通过它询问修复被接受或拒绝的原因、观察到的设置情况及未解决的证据。
--jsonDownload run artifacts manually
手动下载运行工件
Prefer MCP for ordinary inspection because downloads and
extracts cloud attempts automatically. Use the API when MCP output is missing,
when raw files must be grepped, or when independently verifying the agent.
momentic_get_runThe export endpoint returns run metadata plus one short-lived signed ZIP URL per
attempt; it does not return one combined run ZIP. Download and extract them
without printing the URLs:
bash
RUN_ID="<run-id>"
MOMENTIC_SERVER_URL="${MOMENTIC_SERVER:-https://api.momentic.ai}"
RUN_INSPECTION_DIR="$(mktemp -d)"
curl --fail --silent --show-error \
-H "Authorization: Bearer ${MOMENTIC_API_KEY:?Set MOMENTIC_API_KEY}" \
"${MOMENTIC_SERVER_URL}/v1/runs/${RUN_ID}/export" \
-o "${RUN_INSPECTION_DIR}/export.json"
jq '.run' "${RUN_INSPECTION_DIR}/export.json" \
> "${RUN_INSPECTION_DIR}/metadata.json"
jq -r '.attemptZipUrls | to_entries[] | select(.value != null) |
[(.key + 1), .value] | @tsv' "${RUN_INSPECTION_DIR}/export.json" |
while IFS=$'\t' read -r attempt url; do
mkdir -p "${RUN_INSPECTION_DIR}/attempts/${attempt}"
curl --fail --silent --show-error "$url" \
-o "${RUN_INSPECTION_DIR}/attempt-${attempt}.zip"
unzip -q "${RUN_INSPECTION_DIR}/attempt-${attempt}.zip" \
-d "${RUN_INSPECTION_DIR}/attempts/${attempt}"
doneInspect , each attempt's , screenshots, DOM
snapshots, console logs, HAR logs, video, resource usage, and crash archives.
Keep attempt numbers intact. Do not commit the export response, run artifacts,
signed URLs, credentials, or customer data. Remove the temporary directory when
the investigation is complete.
metadata.jsonmetadata.json常规检查优先使用MCP,因为会自动下载并提取云端尝试记录。当MCP输出缺失、需要搜索原始文件或独立验证Agent时,使用API。
momentic_get_run导出端点返回运行元数据及每个尝试记录的短期签名ZIP URL;不返回合并的运行ZIP文件。下载并提取文件时不要打印URL:
bash
RUN_ID="<run-id>"
MOMENTIC_SERVER_URL="${MOMENTIC_SERVER:-https://api.momentic.ai}"
RUN_INSPECTION_DIR="$(mktemp -d)"
curl --fail --silent --show-error \
-H "Authorization: Bearer ${MOMENTIC_API_KEY:?Set MOMENTIC_API_KEY}" \
"${MOMENTIC_SERVER_URL}/v1/runs/${RUN_ID}/export" \
-o "${RUN_INSPECTION_DIR}/export.json"
jq '.run' "${RUN_INSPECTION_DIR}/export.json" \
> "${RUN_INSPECTION_DIR}/metadata.json"
jq -r '.attemptZipUrls | to_entries[] | select(.value != null) |
[(.key + 1), .value] | @tsv' "${RUN_INSPECTION_DIR}/export.json" |
while IFS=$'\t' read -r attempt url; do
mkdir -p "${RUN_INSPECTION_DIR}/attempts/${attempt}"
curl --fail --silent --show-error "$url" \
-o "${RUN_INSPECTION_DIR}/attempt-${attempt}.zip"
unzip -q "${RUN_INSPECTION_DIR}/attempt-${attempt}.zip" \
-d "${RUN_INSPECTION_DIR}/attempts/${attempt}"
done检查、每个尝试记录的、截图、DOM快照、控制台日志、HAR日志、视频、资源使用情况及崩溃归档。保留尝试编号不变。不要提交导出响应、运行工件、签名URL、凭证或客户数据。调查完成后删除临时目录。
metadata.jsonmetadata.jsonBackground
背景知识
Test run result structure
测试运行结果结构
When momentic tests are run via the CLI, the results are stored in a "run group". The data for this run group is stored in a single directory within the momentic project. By default, the directory is called , but can be changed in momentic project settings or on a single run of a run group. The run group results folder has the following structure:
test-resultstest-results/
├── metadata.json data about the run group, including git metadata and timing info.
└── runs/ On zip for each test run in the run group.
├── <runId_1>.zip a zipped run directory containing data about this specific test run. Follows the structure described below.
└── <runId_2>.zipWhen unzipped, run directories have the following structure:
<runId>/
├── metadata.json run-level metadata.
└── attempts/<n>/ one folder per attempt (1-based n).
├── metadata.json attempt outcome and step results.
├── console.json optional browser console output.
└── assets/
├── <snapshotId>.jpeg before/after screenshot for each step (see attempt metadata.json for snapshot ID).
├── <snapshotId>.html before/after DOM snapshot for each step (see attempt metadata.json for snapshot ID).
├── har-pages.log HAR pages (ndjson).
├── har-entries.log HAR network entries (ndjson).
├── resource-usage.ndjson CPU/memory samples taken during the attempt.
├── <videoName> video recording (when video recording is enabled).
└── browser-crash.zip browser crash dump (only present on crash).When getting run results via the momentic MCP, tools such as will return links to the MCP working directory (default ). This directory will contain unzipped run result folders, following the structure above, named .
momentic_get_run.momentic-mcprun-result-<runId>通过CLI运行Momentic测试时,结果存储在“运行组”中。运行组的数据存储在Momentic项目内的单个目录中。默认目录名为,可在Momentic项目设置或单次运行组时修改。运行组结果目录结构如下:
test-resultstest-results/
├── metadata.json 运行组相关数据,包括Git元数据和时间信息。
└── runs/ 运行组中每个测试运行的ZIP文件。
├── <runId_1>.zip 包含该测试运行具体数据的压缩目录,结构如下所述。
└── <runId_2>.zip解压后,运行目录结构如下:
<runId>/
├── metadata.json 运行级元数据。
└── attempts/<n>/ 每个尝试记录对应一个文件夹(n从1开始)。
├── metadata.json 尝试结果和步骤结果。
├── console.json 可选的浏览器控制台输出。
└── assets/
├── <snapshotId>.jpeg 每个步骤的前后截图(查看尝试记录metadata.json获取快照ID)。
├── <snapshotId>.html 每个步骤的前后DOM快照(查看尝试记录metadata.json获取快照ID)。
├── har-pages.log HAR页面(ndjson格式)。
├── har-entries.log HAR网络请求记录(ndjson格式)。
├── resource-usage.ndjson 尝试过程中的CPU/内存采样数据。
├── <videoName> 视频录制(启用视频录制时存在)。
└── browser-crash.zip 浏览器崩溃转储(仅崩溃时存在)。通过Momentic MCP获取运行结果时,等工具会返回指向MCP工作目录(默认)的链接。该目录包含解压后的运行结果文件夹,命名为,结构如上所述。
momentic_get_run.momentic-mcprun-result-<runId>Element locators
元素定位器
Certain step types that interact with elements have a "target" property, or locator, that specifies which element the step should interact with.
某些与元素交互的步骤类型具有“target”属性(即定位器),用于指定步骤应交互的元素。
Locator caches
定位器缓存
Locators identify elements by sending the page state html/xml to an llm as well as a screenshot. The llm identifies which element on the page the user is referring to. Momentic will attempt to "cache" the answer from the llm so that future runs don't require AI calls. On future runs, the page state is checked against the cached element to determine whether the element is still usable, or the page has changed enough such that another AI call is required.
A locator cache can bust for a variety of reasons:
- the element description has changed, in which case we'll always bust the cache
- the cached element could not be located in the current page state
- the cached element was located in the page state, but fails certain checks specified on the cache entry, such as requiring a certain position, shape, or content.
You can find the on the property in the results for a given step, but only when you explicitly request . The property is also listed on the results, showing the full cache saved for that element.
cacheBustReasontraceincludeTrace=truecache定位器通过将页面状态HTML/XML及截图发送给LLM来识别元素,LLM会识别用户指向的页面元素。Momentic会尝试缓存LLM的响应,以便后续运行无需调用AI。后续运行时,会检查当前页面状态与缓存元素是否匹配,判断元素是否仍可用,或页面变化是否需要再次调用AI。
定位器缓存失效的原因包括:
- 元素描述已变更,此时缓存必然失效
- 缓存元素无法在当前页面状态中找到
- 缓存元素可在当前页面状态中找到,但未通过缓存条目指定的某些检查,如位置、形状或内容要求
可在步骤结果的属性中找到,但需显式请求。结果中还会列出属性,显示该元素的完整缓存内容。
tracecacheBustReasonincludeTrace=truecacheIdentifying bad caches
识别无效缓存
Sometimes the element that was cached is not the element that the user intended to target. This can cause failures or unexpected behaviors in tests. In these cases, it helps to verify exactly why the wrong cache was saved in the first place. Only request for these cache-debugging cases or when you suspect incorrect Momentic execution data. Use the property of the on the incorrect cache to get the details of the original run, calling with this runId. This will return the run where the cache target was updated.
includeTrace=truerunIdtargetUpdateLoggerTagsmomentic_get_run有时缓存的元素并非用户预期的目标元素,这会导致测试失败或行为异常。此时需验证错误缓存最初被保存的原因。仅在调试缓存或怀疑Momentic执行数据错误时,才请求。使用错误缓存的中的属性获取原始运行记录详情,调用传入该runId,即可返回缓存目标被更新的运行记录。
includeTrace=truetargetUpdateLoggerTagsrunIdmomentic_get_runModule caching
模块缓存
Cached modules skip executing their steps when the module cache key and resolved inputs are unchanged, and reuse the cached return value from the module's last step.
Authentication modules can also save and restore browser auth state from the module cache, including cookies, localStorage, and IndexedDB. They may use a page-content check after restoring auth state to decide whether the cache is still valid.
当模块缓存键和解析输入未变更时,缓存模块会跳过执行步骤,重用模块最后一步的缓存返回值。
认证模块还可从模块缓存中保存和恢复浏览器认证状态,包括Cookie、localStorage和IndexedDB。恢复认证状态后,模块可能通过页面内容检查判断缓存是否仍有效。
File uploads
文件上传
A file upload step prepares one file for the next native file picker, so it must run before the action that opens the picker.
Sources can be remote URLs, references to earlier downloads, CLI-local paths, or uploaded user files. The step can also override the presented filename, and Momentic wires the prepared file into the browser's file chooser handling.
file://文件上传步骤会为下一个原生文件选择器准备一个文件,因此必须在打开选择器的操作之前运行。
文件来源可以是远程URL、协议引用的早期下载文件、CLI本地路径或用户上传的文件。该步骤还可覆盖显示的文件名,Momentic会将准备好的文件关联到浏览器的文件选择器处理逻辑中。
file://Using past runs
使用历史运行记录
Past runs are comparison evidence, not a substitute for reconstructing the current run. Use them when the current run does not answer:
- When did this test start failing?
- What differed vs the last passing run?
- Did the same action behave differently on an earlier run?
- Is this a test weakness, an application change, a real application bug, or a temporary slowdown?
Use step results and screenshots on past runs to answer these questions. Do NOT rely only on summaries from or to understand what happened in a test run. Look at the specific run details, including step results and screenshots, before citing a past run as evidence.
momentic_get_runmomentic_list_runsWhen looking at past runs, use the following workflow:
- Call the tool to identify the runs you want more detail on. Always pass
momentic_list_runswhen it exists on the run in question. Omit it when you need runs from other branches.gitBranchName - Call for that specific run to get the run details.
momentic_get_run - Call for the same step/container or closest equivalent you are comparing, especially for screenshots.
momentic_get_step_result
When past runs are irrelevant because the current run already proves the root cause, say that briefly instead of forcing historical evidence.
历史运行记录仅作为对比证据,不能替代对当前运行记录的重构。仅在当前运行记录无法解答以下问题时使用:
- 该测试何时开始失败?
- 与上次成功运行记录有何差异?
- 同一操作在早期运行记录中的表现是否不同?
- 这是测试缺陷、应用程序变更、真实应用程序bug还是临时性能下降?
使用历史运行记录的步骤结果和截图解答这些问题。不要仅依赖或的摘要来理解测试运行情况。在引用历史运行记录作为证据前,需查看具体的运行详情,包括步骤结果和截图。
momentic_get_runmomentic_list_runs查看历史运行记录时,遵循以下工作流:
- 调用工具确定需要查看详情的运行记录。当运行记录存在
momentic_list_runs时,务必传入该参数。仅在需要查看其他分支的运行记录时才省略。gitBranchName - 调用获取该运行记录的详情。
momentic_get_run - 调用获取要对比的相同步骤/容器或最接近的等效项,尤其是截图。
momentic_get_step_result
当当前运行记录已能证明根本原因,历史运行记录无关时,可简要说明,无需强行提供历史证据。
Multi-attempt runs
多尝试运行记录
When shows a passing run with , treat it as a partial failure worth investigating, not a clean passing run. Use the parameter to retrieve earlier failed attempt results for that run to understand what was going wrong before the retry succeeded.
momentic_list_runsattempts > 1attemptNumber当显示的成功运行记录时,需将其视为值得调查的部分失败,而非完全成功的运行记录。使用参数获取该运行记录的早期失败尝试结果,了解重试成功前存在的问题。
momentic_list_runsattempts > 1attemptNumberFlakiness and intermittent failures
抖动和间歇性故障
- Call a test flaky only when the same app and test behavior fails intermittently. One failure is insufficient; rule out application and test changes using run evidence before attributing the pattern to flakiness.
- 仅当相同应用程序和测试行为间歇性失败时,才称测试存在抖动。单次失败不足以判定;需通过运行证据排除应用程序和测试变更后,再将该模式归因于抖动。
Test temporality
测试时效性
- Any past results may not necessarily match today’s test file. The test may have changed, meaning the result was on a different version of the test.
- You can call to help you determine if the test itself changed between runs, although note that this tool returns a summary of each test step. If you suspect that specific details on certain steps have changed between test runs, full step details are included in the response from
get_test_steps_for_run; only requestmomentic_get_step_resultwhen those fields and screenshots still are not enough.includeTrace=true
- 任何历史结果可能与当前测试文件不匹配。测试可能已变更,意味着历史结果基于不同版本的测试。
- 可调用帮助判断运行记录间测试是否变更,但需注意该工具返回的是每个测试步骤的摘要。若怀疑特定步骤的具体细节在运行记录间发生变化,
get_test_steps_for_run的响应中包含完整步骤详情;仅当这些字段和截图仍不足以解答时,才请求momentic_get_step_result。includeTrace=true
Common failure modes to watch for
需要关注的常见故障模式
- A setup module appears to pass but leaves the wrong page, overlay, filter, search, selected row, or side peek open. Classify from the step/container that left the bad postcondition, not only from the next step that failed.
- A click reports success and targets the intended element, but the application does not transition to the intended state. Verify the post-state; do not assume the click worked because the locator was correct.
- A weak global assertion such as "page contains X" passes because unrelated text on the page matches. The next step may then type or click in the wrong context.
- A type step without a specific target can silently type nowhere useful if the preceding action failed to focus the intended field.
- A locator or cache can be technically valid but semantically wrong. Check the interacted element and, for bad caches, inspect the original cache-update run from .
targetUpdateLoggerTags.runId - A recovered step can hide the first failure. Inspect failed child steps inside recovered modules and compare the recovered final state to the next step's expected baseline.
- A timeout is not automatically . First rule out missing data, wrong page state, changed app flow, bad locator/assertion, and setup failure.
INFRA
- 设置模块看似通过,但留下错误页面、覆盖层、过滤器、搜索条件、选中行或侧边弹窗未关闭。需从产生异常后置条件的步骤/容器进行分类,而非仅从后续失败步骤分类。
- 点击操作报告成功且目标为预期元素,但应用程序未切换到预期状态。需验证后置状态;不要因定位器正确就假设点击操作成功。
- 宽泛的全局断言(如“页面包含X”)因页面上的无关文本而通过,导致后续步骤在错误上下文中输入或点击。
- 无特定目标的输入步骤,若前置操作未成功聚焦预期字段,可能会在无效位置静默输入。
- 定位器或缓存技术上有效,但语义错误。检查交互元素;对于无效缓存,从查看原始缓存更新运行记录。
targetUpdateLoggerTags.runId - 已恢复步骤可能掩盖首次失败。检查已恢复模块内的失败子步骤,对比恢复后的最终状态与后续步骤的预期基准状态。
- 超时并不自动归类为。需先排除缺失数据、错误页面状态、应用流程变更、无效定位器/断言及设置失败等原因。
INFRA
Identifying related vs unrelated issues
识别相关与无关问题
- Determine intent from the test name, description, and simplified steps. Treat
failures outside that intent, including most setup/teardown failures, as
unrelated. Relatedness qualifies bugs and changes; it does not change an
failure into another category.
INFRA
- 从测试名称、描述和简化步骤判断意图。将超出该意图的故障(包括大多数设置/清理故障)视为无关问题。相关性用于区分bug和变更,但不会将故障转换为其他类别。
INFRA
Bug vs change
Bug与变更
- Bug: something very clearly went wrong when it should not have, such as an error message appearing. It is obvious just by looking at a single step or two that this is a bug.
- Change: a clear change in the application behavior that you can prove through screenshots.
- Bug:明显不应发生的错误,如出现错误提示。仅查看一两个步骤即可明显判断为bug。
- 变更:可通过截图证明的应用程序行为明确变化。
Recoverability
可恢复性
Along with the category, determine one recoverability value:
- — The failure can be automatically fixed by updating the test itself so that future runs pass.
RECOVERABLE- Examples: an application change that requires a test update; vague locators or assertions that can be rewritten to pass stably.
- — The failure can be recovered for this specific run without persisting a test change.
ONE_TIME_RECOVERABLE- Examples: a random modal that can be dismissed without affecting test purpose; a temporary delay where waiting or retrying would likely succeed.
- — The failure cannot be automatically addressed and requires manual intervention.
NON_RECOVERABLE- Examples: missing credentials; missing local files required for upload; outages likely caused by third-party systems where test steps cannot fix the issue.
除类别外,需确定一个可恢复性值:
- — 可通过更新测试本身自动修复故障,使后续运行通过。
RECOVERABLE- 示例:应用程序变更需要更新测试;模糊定位器或断言可重写为稳定通过的形式。
- — 可针对该特定运行记录恢复故障,无需持久化测试变更。
ONE_TIME_RECOVERABLE- 示例:可在不影响测试目的的情况下关闭随机模态框;临时延迟可通过等待或重试解决。
- — 无法自动处理故障,需手动干预。
NON_RECOVERABLE- 示例:缺失凭证;文件上传步骤所需的本地文件缺失;第三方系统中断导致的故障,无法通过测试步骤修复。
Formal classification output
正式分类输出
- Exactly one category id — no new labels, no multi-label.
- Ground your decision in data. Be sure that you've fully investigated the run before assigning the category.
- Prefer human-readable references over UUIDs when the step/module can be identified colloquially: ,
module create-subpage-under-parent-page,the last invocation of module <name>,substep 4 (0-indexed), etc. Tool calls still require exact IDs, but final reasoning should be readable.the failed setup assertion - When referencing past runs in final output, use clickable Momentic URLs rather than bare UUIDs: . Do not shorten UUIDs inside those URLs.
https://app.momentic.ai/runs/<runId> - The reasoning must include the earliest divergent step/container and the broken postcondition it produced, not just the final failing step.
text
Reasoning: <a few sentences tied to the earliest divergence, screenshots/traces, past runs if used, and test intent>
Category: <one id from the list>
Recoverable: <RECOVERABLE | ONE_TIME_RECOVERABLE | NON_RECOVERABLE>
Confidence: <high | medium | low>Confidence levels:
- — direct evidence, such as a clear screenshot of a label change or crash
high - — strong inference from multiple signals but no single conclusive screenshot or data point
medium - — ambiguous evidence; the classification required significant inference or the root cause is unclear
low
- 精确指定一个类别ID — 不要新增标签,不要多标签。
- 基于数据做出决策。在分配类别前,务必完成对运行记录的全面调查。
- 当步骤/模块可通俗识别时,优先使用人类可读的引用而非UUID:、
module create-subpage-under-parent-page、模块<name>的最后一次调用、子步骤4(从0开始计数)等。工具调用仍需精确ID,但最终推理过程应易于阅读。失败的设置断言 - 在最终输出中引用历史运行记录时,使用可点击的Momentic URL而非纯UUID:。不要缩短URL中的UUID。
https://app.momentic.ai/runs/<runId> - 推理过程必须包含最早出现偏差的步骤/容器及其产生的损坏后置条件,而非仅最终失败步骤。
text
Reasoning: <几句与最早偏差、截图/跟踪信息、所用历史运行记录及测试意图相关的描述>
Category: <列表中的一个ID>
Recoverable: <RECOVERABLE | ONE_TIME_RECOVERABLE | NON_RECOVERABLE>
Confidence: <high | medium | low>置信度级别:
- — 直接证据,如标签变更或崩溃的清晰截图
high - — 多个信号的强推论,但无单一决定性截图或数据点
medium - — 证据模糊;分类需大量推论或根本原因不明确
low
Category ids
类别ID
Use these strings verbatim:
- — The run had no failures; all attempts passed.
NO_FAILURE - — The test is out of date because the application's flow or UI has changed; updating the test to match the new behavior would permanently fix the failure.
APPLICATION_CHANGE - — Something clearly went wrong in the application that shouldn't have, such as an error message appearing or expected content failing to render.
BUG - — The test can be permanently updated to prevent the failure while still validating its original intent, and you can recommend a specific authorship change such as adding or modifying a step, rewriting a vague assertion, or making a locator description more specific. If you cannot name a concrete change, choose a different category. Timeouts, slow page loads, and any failure whose recommended fix is to "wait longer" or to increase a timeout are NOT authorship issues — those are
TEST_AUTHORSHIP, even when the test could technically be edited to wait longer.INFRA- Examples: race conditions that can be fixed by adding or modifying steps other than waits/timeouts (e.g. replacing a "type with pressEnter" step with an explicit "select from list" step so the test no longer races the application); vague assertions or locator descriptions that can be rewritten to be more specific.
- — Missing test data or files necessary to run the test, where the fix requires user action outside of the test itself.
TEST_SETUP- Examples: missing file for a file upload step; missing or incorrect credentials needed by the test.
- — The failure was unrelated to the application or application code and was caused by an infrastructure outage, long load times, or some other issue due to outside factors.
INFRA- Examples: browser crash; high resource usage; rate limiting; a step or assertion that timed out waiting for the page or application to reach a slow-but-eventual state.
- — Some issue occurred with the execution of the test or Momentic data was incorrect (e.g. cache is wrong, global locator redirect did something weird, AI hallucinations).
MOMENTIC_ISSUE- Examples: unexpected behavior when viewing the run trace; the AI clearly misread or hallucinated data that is unambiguous in the screenshot, and no reasonable test alternative exists to avoid the AI step.
- — The failure doesn't fit any of the other categories.
OTHER
请严格使用以下字符串:
- — 运行记录无故障;所有尝试均通过。
NO_FAILURE - — 测试已过时,因应用程序流程或UI已变更;更新测试以匹配新行为可永久修复故障。
APPLICATION_CHANGE - — 应用程序出现明显不应发生的错误,如出现错误提示或预期内容未渲染。
BUG - — 可通过永久更新测试防止故障,同时仍验证其原始意图,且可推荐具体的编写变更,如添加或修改步骤、重写模糊断言、使定位器描述更具体。若无法明确具体变更,请选择其他类别。超时、页面加载缓慢及建议修复为“等待更长时间”或增加超时的故障不属于编写问题 — 这些属于
TEST_AUTHORSHIP,即使测试技术上可编辑为延长等待时间。INFRA- 示例:可通过添加或修改步骤(而非等待/超时)修复的竞争条件(如将“输入并按回车”步骤替换为显式“从列表选择”步骤,使测试不再与应用程序竞争);可重写为更具体的模糊断言或定位器描述。
- — 运行测试所需的测试数据或文件缺失,修复需在测试本身之外进行用户操作。
TEST_SETUP- 示例:文件上传步骤所需文件缺失;测试所需凭证缺失或错误。
- — 故障与应用程序或应用程序代码无关,由基础设施中断、加载时间过长或其他外部因素导致。
INFRA- 示例:浏览器崩溃;资源使用率过高;速率限制;步骤或断言因等待页面或应用程序达到缓慢但最终的状态而超时。
- — 测试执行出现问题或Momentic数据错误(如缓存错误、全局定位器重定向异常、AI幻觉)。
MOMENTIC_ISSUE- 示例:查看运行跟踪时出现意外行为;AI明显误读或幻觉出截图中明确的数据,且无合理的测试替代方案可避免该AI步骤。
- — 故障不符合上述任何类别。
OTHER