Back to Details

cherry-regression-test

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Cherry Regression Test

Cherry 回归测试

Run deterministic Playwright E2E tests against one driver-owned Cherry Studio process per platform. The tested product includes Chat, Agents, MCP, Skills, knowledge bases, translation, image generation, and code tools; an LLM test agent does not control the test run.
在每个平台上，针对一个由回归测试 driver 管理的 Cherry Studio 进程运行确定性的 Playwright E2E 测试。被测产品包括 Chat、Agents、MCP、Skills、 知识库、翻译、图像生成和代码工具；LLM 测试 Agent 不控制测试执行。

CI contract

CI 契约

Use 
.github/workflows/e2e-regression-test.yml
as the entry point. It:
  1. Resolves a trusted branch or release tag.
  2. Initializes an isolated directory under the GitHub runner temporary folder.
  3. Installs the application and the code tools under test.
  4. Launches one owned Electron process with CDP enabled.
  5. Runs 
    tests/e2e/regression/phases/*.test.ts
    in the manifest's phase order.
  6. Continues after a failed phase so later results are still collected.
  7. Produces English platform and aggregate reports, then enforces the verdict.
  8. Stops only the Electron process recorded in the isolated run directory.
Do not add an LLM tool loop, MCP control server, turn limit, or a second Electron launch for each test. A restart is allowed only where the case contract explicitly verifies persistence or switches from the clean startup profile to the authenticated shared profile.
使用 
.github/workflows/e2e-regression-test.yml
作为入口。它：
  1. 确定一个受信任的分支或发布标签。
  2. 在 GitHub runner 的临时文件夹下初始化一个隔离目录。
  3. 安装被测应用和代码工具。
  4. 启动一个由该 driver 管理的 Electron 进程，并启用 CDP。
  5. 按 manifest 中的阶段顺序运行 
    tests/e2e/regression/phases/*.test.ts
    。
  6. 阶段失败后仍继续执行，以便收集后续结果。
  7. 生成英文的平台报告和汇总报告，然后据此执行最终判定。
  8. 仅停止隔离运行目录中记录的 Electron 进程。
不要添加 LLM 工具循环、MCP 控制服务器、轮次限制，也不要为每个测试进行第二次 Electron 启动。仅当用例契约明确验证持久化，或从全新启动 profile 切换到 已认证的共享 profile 时，才允许重启。

Configuration

配置

The workflow reads these repository variables and secrets:
  • CHERRY_TEST_CUSTOM_PROVIDER_BASE_URL
  • CHERRY_TEST_CUSTOM_PROVIDER_ANTHROPIC_BASE_URL
  • CHERRY_TEST_CUSTOM_PROVIDER_API_KEY
  • CHERRY_TEST_CUSTOM_PROVIDER_CHAT_MODEL
  • CHERRY_TEST_CUSTOM_PROVIDER_EMBEDDING_BASE_URL
  • CHERRY_TEST_CUSTOM_PROVIDER_EMBEDDING_API_KEY
  • CHERRY_TEST_CUSTOM_PROVIDER_EMBEDDING_MODEL
  • CHERRY_TEST_CHERRYIN_CHAT_MODEL
  • CHERRY_TEST_CHERRYIN_IMAGE_MODEL
  • CHERRY_TEST_CHERRYIN_ACCOUNT
  • CHERRY_TEST_CHERRYIN_PASSWORD
The custom chat provider requires both URLs: 
CHERRY_TEST_CUSTOM_PROVIDER_BASE_URL
fills OpenAI, and 
CHERRY_TEST_CUSTOM_PROVIDER_ANTHROPIC_BASE_URL
fills Anthropic. Both endpoints share 
CHERRY_TEST_CUSTOM_PROVIDER_API_KEY
.
Chat and embedding providers are independent. Never print literal credentials, write them to fixtures, attach them to Playwright artifacts, or pass them to an unrelated action.
工作流读取以下仓库变量和 secrets：
  • CHERRY_TEST_CUSTOM_PROVIDER_BASE_URL
  • CHERRY_TEST_CUSTOM_PROVIDER_ANTHROPIC_BASE_URL
  • CHERRY_TEST_CUSTOM_PROVIDER_API_KEY
  • CHERRY_TEST_CUSTOM_PROVIDER_CHAT_MODEL
  • CHERRY_TEST_CUSTOM_PROVIDER_EMBEDDING_BASE_URL
  • CHERRY_TEST_CUSTOM_PROVIDER_EMBEDDING_API_KEY
  • CHERRY_TEST_CUSTOM_PROVIDER_EMBEDDING_MODEL
  • CHERRY_TEST_CHERRYIN_CHAT_MODEL
  • CHERRY_TEST_CHERRYIN_IMAGE_MODEL
  • CHERRY_TEST_CHERRYIN_ACCOUNT
  • CHERRY_TEST_CHERRYIN_PASSWORD
自定义聊天 provider 必须同时提供两个 URL：
CHERRY_TEST_CUSTOM_PROVIDER_BASE_URL
用于填充 OpenAI 配置，
CHERRY_TEST_CUSTOM_PROVIDER_ANTHROPIC_BASE_URL
用于填充 Anthropic 配置。 两个 endpoint 共用 
CHERRY_TEST_CUSTOM_PROVIDER_API_KEY
。
Chat 和 embedding provider 相互独立。绝不打印明文凭据、将其写入 fixtures、 将其附加到 Playwright artifacts，或传递给无关 action。

Test organization

测试组织

Read scenario organization and the controller contract before making changes.
Register each case from the manifest:
ts
test(...caseDefinition('S-01'), async ({ mainWindow }) => {
  // Assert the user-visible outcome.
})
cases.ts
owns case IDs, titles, task tags, phases, and capability requirements. The workflow accepts a task ID and delegates selection to the controller. Prefer accessible roles, labels, placeholders, test IDs, and visible text. Native dialogs and cross-application interactions must use the repository-owned helpers in 
systemAutomation.ts
.
Record assertions in Playwright, not prose. The custom reporter writes case and phase status into 
run.json
and feeds the English Markdown/JUnit reports. The fixture saves failure screenshots. Executor errors and interrupted phases block a passing verdict. Do not enable Playwright Trace for credential-bearing tests because action parameters can expose secrets. A passing result does not depend on a model's judgment.
修改前，请阅读 场景组织 和 控制器契约。
在 manifest 中注册每个用例：
ts
test(...caseDefinition('S-01'), async ({ mainWindow }) => {
  // Assert the user-visible outcome.
})
cases.ts
负责管理用例 ID、标题、任务标签、阶段和能力要求。 工作流接受一个 task ID，并将用例选择委托给 controller。 优先使用可访问的 roles、labels、placeholders、test IDs 和可见文本。 原生对话框和跨应用交互必须使用 
systemAutomation.ts
中由仓库提供的 helpers。
在 Playwright 中记录断言，而不是用文字描述。自定义 reporter 将用例和阶段状态写入 
run.json
，并为英文 Markdown/JUnit 报告提供数据。fixture 会保存失败截图。Executor 错误和 中断的阶段都会阻止通过判定。对于包含凭据的测试，不要启用 Playwright Trace，因为 action 参数可能暴露 secrets。是否通过不依赖模型的判断。

Focused execution

定向执行

With an initialized run directory and its owned Electron process running:
bash
pnpm exec tsx scripts/e2e/regression/cli.ts run-phase \
  --run-dir /absolute/run-directory --phase basicFeatures
The task selected when initializing the run determines which cases execute. For a Notes-only run, initialize with 
--task notes
. Enumeration is read-only: set 
CHERRY_TEST_RUN_DIR
to an absolute path, but no initialized run directory or running Electron process is required because 
--list
does not execute fixtures.
bash
CHERRY_TEST_RUN_DIR=/tmp/cherry-regression-list pnpm test:e2e:regression --list
Do not call the regression cleanup command for an Electron instance owned by 
cherry-electron-dev
; cleanup is only for an app record created by this driver.
在已初始化的运行目录及其管理的 Electron 进程正在运行时：
bash
pnpm exec tsx scripts/e2e/regression/cli.ts run-phase \
  --run-dir /absolute/run-directory --phase basicFeatures
初始化 run 时选定的 task 决定执行哪些用例。对于仅运行 Notes 的场景，初始化时使用 
--task notes
。用例枚举是只读操作：将 
CHERRY_TEST_RUN_DIR
设置为绝对路径，但由于 
--list
不会执行 fixtures，因此无需初始化运行目录，也无需运行中的 Electron 进程。
bash
CHERRY_TEST_RUN_DIR=/tmp/cherry-regression-list pnpm test:e2e:regression --list
不要对 
cherry-electron-dev
所拥有的 Electron 实例调用回归清理命令；清理操作仅适用于由本 driver 创建的应用记录。

Verification when changing the framework

修改框架时的验证

Run the focused script suite and enumerate Playwright cases:
bash
pnpm exec vitest run --project scripts scripts/e2e/regression
CHERRY_TEST_RUN_DIR=/tmp/cherry-regression-list \
  pnpm test:e2e:regression --list
pnpm typecheck:e2e
pnpm test:lint
Do not use 
pnpm test
or 
pnpm build:check
for this focused workflow change.
运行定向脚本测试套件并枚举 Playwright 用例：
bash
pnpm exec vitest run --project scripts scripts/e2e/regression
CHERRY_TEST_RUN_DIR=/tmp/cherry-regression-list \
  pnpm test:e2e:regression --list
pnpm typecheck:e2e
pnpm test:lint
对于此次定向工作流变更，不要使用 
pnpm test
或 
pnpm build:check
。