analyze-selective-testing

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Analyze Selective Testing

分析选择性测试

Quick Start

快速开始

Run
```
tuist test list --json
```
to find test runs.

Run

tuist test xcode target list <test-run-id> --json

to see per-target selective testing status.

If effectiveness dropped, compare targets between a known-good run and the regressed run.

运行
```
tuist test list --json
```
查找测试运行记录。
运行
```
tuist test xcode target list <test-run-id> --json
```
查看每个目标的选择性测试状态。
如果有效性下降，对比已知正常运行记录与出现回归的运行记录中的目标情况。

Step 1: Resolve Test Runs

步骤1：确定测试运行记录

Find the test run to analyze

找到要分析的测试运行记录

List recent test runs, optionally filtering by branch:

bash

tuist test list --json --page-size 10
tuist test list --git-branch feature-x --json --page-size 5

Get basic info for a specific test run:

bash

tuist test show <test-run-id> --json

Use this to confirm the run's branch, status, scheme, and environment (

xcode_version

macos_version

列出近期的测试运行记录，可选择按分支过滤：

bash

tuist test list --json --page-size 10
tuist test list --git-branch feature-x --json --page-size 5

获取特定测试运行记录的基本信息：

bash

tuist test show <test-run-id> --json

使用此命令确认运行记录的分支、状态、方案以及环境信息（

xcode_version

、

macos_version

）。

Defaults

默认规则

If no test run is specified, use the most recent CI test run on the current branch.
For comparison, use the most recent CI test run on the project's default branch.

如果未指定测试运行记录，则使用当前分支上最近一次CI测试运行记录。
用于对比时，使用项目默认分支上最近一次CI测试运行记录。

Step 2: Drill Into Selective Testing Targets

步骤2：深入分析选择性测试目标

List per-target selective testing data for a test run:

bash

tuist test xcode target list <test-run-id> --json

Each target includes:

name: The test target name
hit_status:
```
miss
```
(ran),
```
local
```
(skipped via local cache), or
```
remote
```
(skipped via remote cache)
hash: The selective testing hash for this target

Filter by status to focus your analysis:

bash

tuist test xcode target list <test-run-id> --hit-status miss --json
tuist test xcode target list <test-run-id> --hit-status local --json
tuist test xcode target list <test-run-id> --hit-status remote --json

列出某一测试运行记录中每个目标的选择性测试数据：

bash

tuist test xcode target list <test-run-id> --json

每个目标包含：

name：测试目标名称
hit_status：
```
miss
```
（已执行）、
```
local
```
（通过本地缓存跳过）或
```
remote
```
（通过远程缓存跳过）
hash：该目标的选择性测试哈希值

按状态过滤以聚焦分析：

bash

tuist test xcode target list <test-run-id> --hit-status miss --json
tuist test xcode target list <test-run-id> --hit-status local --json
tuist test xcode target list <test-run-id> --hit-status remote --json

Assess effectiveness

评估有效性

Count the targets by status:

effectiveness = (local_hits + remote_hits) / total_targets * 100

Effectiveness	Verdict
60-100%	Healthy — most unchanged tests are being skipped
30-60%	Moderate — some cache invalidation occurring, worth investigating
0-30%	Low — likely a core dependency changed or the cache was invalidated
0%	Cold cache — first run, environment change, or full invalidation

按状态统计目标数量：

effectiveness = (local_hits + remote_hits) / total_targets * 100

有效性	结论
60-100%	健康——大多数未变更的测试已被跳过
30-60%	一般——存在部分缓存失效情况，值得调查
0-30%	低——核心依赖可能已变更或缓存已失效
0%	冷缓存——首次运行、环境变更或完全失效

Step 3: Diagnose Regressions

步骤3：诊断回归问题

If effectiveness dropped, investigate:

如果有效性下降，进行以下调查：

Check for global invalidation

检查全局失效情况

If nearly all targets show as

miss

, the entire selective testing cache was invalidated. Common causes:

Cause	How to verify
Xcode version change	Compare `xcode_version` between good and bad runs via `tuist test show`
CI environment change	Compare `macos_version` , `model_identifier` between runs
Project graph or dependency change	Check git history for changes to project manifests, dependency versions, or target configurations
Core dependency change	A widely-depended-on target changed, invalidating all its dependents

Note: Tuist CLI version upgrades rarely cause hash invalidation — the hash version is not updated on every release.

如果几乎所有目标都显示为

miss

，则整个选择性测试缓存已失效。常见原因：

原因	验证方式
Xcode版本变更	通过 `tuist test show` 对比正常与异常运行记录的 `xcode_version`
CI环境变更	对比运行记录的 `macos_version` 、 `model_identifier`
项目图或依赖变更	检查git历史中项目清单、依赖版本或目标配置的变更
核心依赖变更	被广泛依赖的目标发生变更，导致所有依赖它的目标失效

注意：Tuist CLI版本升级很少导致哈希值失效——哈希版本不会在每次发布时更新。

Check for cascade invalidation

检查级联失效情况

If only some targets show as

miss

, a dependency change likely cascaded:

Run

tuist test xcode target list <good-run-id> --json

and

tuist test xcode target list <bad-run-id> --json

Match targets by name — identify those that changed from
```
local
```
/
```
remote
```
to
```
miss
```
.
Compare hashes for changed targets — if a target's hash differs, its sources or dependencies changed.
Look for a common dependency among the invalidated targets.

如果只有部分目标显示为

miss

，则可能是依赖变更引发的级联效应：

运行

tuist test xcode target list <good-run-id> --json

和

tuist test xcode target list <bad-run-id> --json

。

按名称匹配目标——找出从
```
local
```
/
```
remote
```
变为
```
miss
```
的目标。
对比变更目标的哈希值——如果目标哈希值不同，则其源码或依赖已变更。
查找失效目标之间的共同依赖。

Compare two test runs

对比两次测试运行记录

For each target:

Match by
```
name
```
Compare
```
hit_status
```
(hit -> miss = invalidated, miss -> hit = improved)
Compare
```
hash
```
(different hash = target or its dependencies changed)

针对每个目标：

按
```
name
```
匹配
对比
```
hit_status
```
（命中→未命中=失效，未命中→命中=改善）
对比
```
hash
```
（哈希值不同=目标或其依赖已变更）

Step 4: Recommendations

步骤4：建议

Based on the diagnosis:

Environment change: Ensure CI uses consistent Xcode/macOS versions. This is the most common cause of full cache invalidation.
Cascade from dependency change: Consider if the change to the root target was necessary. If the root target is a widely-shared module, consider splitting it.
Cold cache on new branch: Run tests once to warm the cache. Subsequent runs will benefit from selective testing.

根据诊断结果：

环境变更：确保CI使用一致的Xcode/macOS版本。这是导致完全缓存失效最常见的原因。
依赖变更引发的级联效应：考虑根目标的变更是否必要。如果根目标是广泛共享的模块，可考虑拆分它。
新分支上的冷缓存：运行一次测试以预热缓存。后续运行将受益于选择性测试。

Summary Format

总结格式

Produce a summary with:

Overall: effectiveness percentage, verdict (healthy/moderate/low/cold)
Target breakdown: targets grouped by hit status
Comparison (if applicable): targets that changed status between runs, with hash diffs
Root cause: what caused the regression
Recommendations: actionable next steps

Example:

Selective Testing Analysis: test run abc123 on feature-x

Effectiveness: 15% (3/20 targets skipped) -- LOW
  Local hits:  2 targets (CoreTests, UtilTests)
  Remote hits: 1 target (NetworkTests)
  Misses:      17 targets

Comparison with baseline (run def456, 75% effectiveness):
  14 targets changed from hit to miss
  All changed targets have different hashes

Root cause: Xcode version changed from 15.2 to 16.0 between runs.
This changed all target hashes, invalidating the entire cache.

Recommendations:
- Align CI Xcode version with the version used in the baseline run.
- If the upgrade is intentional, run tests once to re-warm the cache.

生成包含以下内容的总结：

总体情况：有效性百分比、结论（健康/一般/低/冷缓存）
目标细分：按命中状态分组的目标
对比情况（如适用）：运行记录间状态变更的目标，以及哈希值差异
根本原因：导致回归的原因
建议：可执行的下一步操作

示例：

选择性测试分析：feature-x分支上的测试运行记录abc123

有效性：15%（20个目标中3个被跳过）——低
  本地命中：2个目标（CoreTests、UtilTests）
  远程命中：1个目标（NetworkTests）
  未命中：17个目标

与基准记录（运行记录def456，有效性75%）对比：
  14个目标从命中变为未命中
  所有变更目标的哈希值均不同

根本原因：两次运行记录间Xcode版本从15.2变更为16.0。
这导致所有目标哈希值变更，使整个缓存失效。

建议：
- 使CI的Xcode版本与基准记录使用的版本保持一致。
- 如果升级是有意为之，运行一次测试以重新预热缓存。

Done Checklist

完成检查清单

Identified the test run(s) to analyze
Drilled into per-target selective testing status via
```
tuist test xcode target list
```
Diagnosed root cause if effectiveness is low
Compared targets with baseline if regression detected
Provided actionable recommendations

已确定要分析的测试运行记录
通过
```
tuist test xcode target list
```
深入查看每个目标的选择性测试状态
若有效性低，已诊断根本原因
若检测到回归，已与基准记录对比目标情况
已提供可执行的建议