Back to Details

ae-experiment-insight

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

AE Experiment Insight and Diagnosis

AE实验洞察与诊断

Validate the evidence before interpreting the effect. Produce a decision only when the experiment is trustworthy enough to support one.
在解读实验效果前先验证证据。只有当实验足够可信时，才能基于其做出决策。

Platform and reference routing

平台与参考文档指引

Use 
ae-cli
for every AE/TE platform interaction and follow 
references/platform-operations.md
; never substitute another platform access path.
For diagnosis, read 
references/diagnostic-playbook.md
. Before a rollout decision, read 
references/decision-framework.md
.
所有AE/TE平台交互均需使用
ae-cli
，并遵循 
references/platform-operations.md
； 切勿使用其他平台访问路径。
诊断时，请阅读 
references/diagnostic-playbook.md
。 做出推出决策前，请阅读 
references/decision-framework.md
。

Workflow

工作流程

1. Restore experiment context

1. 还原实验上下文

Resolve the exact project and experiment. Collect:
  • hypothesis and intended decision;
  • control and treatment definitions;
  • assignment unit, layer, targeting, traffic, and group allocation;
  • Feature and group values;
  • primary, secondary, and diagnostic metrics;
  • planned MDE, alpha, power, duration, and stopping rule;
  • actual start/end time and material configuration changes.
Without the original hypothesis or success rule, explain observed effects but do not retroactively invent success criteria.
明确具体的项目和实验。收集以下信息：
  • 假设与预期决策；
  • 对照组与处理组定义；
  • 分配单元、分层、定向规则、流量及分组分配；
  • 功能与分组取值；
  • 核心、次要及诊断指标；
  • 计划的MDE、alpha、统计功效、时长及停止规则；
  • 实际起止时间及重要配置变更。
若缺少原始假设或成功规则，仅需解读观测到的效果，不得事后编造成功标准。

2. Check run integrity

2. 检查实验运行完整性

Establish the experiment's actual state, complete runtime periods, and material mid-run Feature, traffic, audience, metric, or identity changes. Treat configuration history as evidence. Evaluate allocation, exposure, sample sufficiency, and design quality through the diagnostic playbook rather than duplicating its rules here.
确认实验的实际状态、完整运行周期，以及运行过程中功能、流量、受众、指标或身份标识的重要变更。将配置历史视为证据。通过诊断手册评估分配、曝光、样本充足性及设计质量，无需在此重复手册中的规则。

3. Run data diagnostics

3. 执行数据诊断

Apply 
diagnostic-playbook.md
in causal order and complete its diagnostic checklist. Use 
scripts/analyze_experiment.py
 for SRM and group comparisons; do not calculate p-values or confidence intervals mentally.
按因果顺序应用
diagnostic-playbook.md
，完成诊断检查清单。使用 
scripts/analyze_experiment.py
进行SRM和分组比较；不得手动计算p值或置信区间。

4. Interpret effects

4. 解读实验效果

For each pre-registered metric, interpret the verified metric contract from the diagnostic playbook and report:
  • verified source event, aggregation, analysis unit, denominator or population, filters, and attribution window;
  • control and treatment values;
  • absolute and relative difference;
  • confidence interval and p-value from deterministic calculation or the verified platform report;
  • achieved sample versus planned sample;
  • relation to MDE and business threshold;
  • statistical and practical significance.
Distinguish:
  • statistically significant and practically valuable;
  • statistically significant but too small to matter;
  • directionally positive but underpowered;
  • no detectable effect within the current precision;
  • significantly negative;
  • invalid or inconclusive because of data quality.
Do not translate 
p > alpha
into “no effect.” State that the current data did not establish an effect and describe the compatible interval.
针对每个预先注册的指标，依据诊断手册中的已验证指标协议进行解读，并报告：
  • 已验证的源事件、聚合方式、分析单元、分母或群体、过滤规则及归因窗口；
  • 对照组与处理组的取值；
  • 绝对差异与相对差异；
  • 通过确定性计算或已验证平台报告得到的置信区间和p值；
  • 实际样本量与计划样本量对比；
  • 与MDE及业务阈值的关系；
  • 统计显著性与实际业务价值显著性。
需区分以下情况：
  • 统计显著且具备实际业务价值；
  • 统计显著但影响过小无实际意义；
  • 趋势正向但统计功效不足；
  • 在当前精度范围内未检测到显著效果；
  • 统计显著且效果负面；
  • 因数据质量问题导致结果无效或无结论。
不得将
p > alpha
翻译为“无效果”。应说明当前数据未证实存在效果，并描述兼容区间。

5. Analyze trends and segments

5. 分析趋势与细分群体

Check trend stability and abnormal dates using complete comparable periods. Apply the trend and segment rules in 
decision-framework.md
.
使用完整的可比周期检查趋势稳定性及异常日期。应用
decision-framework.md
中的趋势与细分群体规则。

6. Diagnose causes

6. 诊断原因

For every suspected cause, provide:
  • observed evidence;
  • inference and uncertainty;
  • competing explanation;
  • the exact platform query or product check that would distinguish them;
  • remediation if confirmed.
Do not produce a list of generic causes detached from evidence. In the final report, express the distinguishing check as a concise user-facing verification action. Include the raw 
ae-cli
command only when the user explicitly asks for commands, an audit trail, or debugging details.
针对每个疑似原因，需提供：
  • 观测到的证据；
  • 推论及不确定性；
  • 竞争性解释；
  • 可区分不同解释的具体平台查询或产品检查方法；
  • 若确认后的补救措施。
不得脱离证据列出通用原因清单。在最终报告中，将区分检查表述为简洁的用户可执行验证操作。仅当用户明确要求命令、审计追踪或调试细节时，才需包含原始
ae-cli
命令。

7. Make the decision

7. 做出决策

Apply 
decision-framework.md
and return one decision supported by the trust assessment, primary effect, duration, trends, segments, and diagnostic results.
Never start, pause, end, change traffic, delete, or roll out an experiment unless the user separately asks for that platform action. Those actions are outside interpretation and require explicit target-and-impact confirmation.
应用
decision-framework.md
，结合可信度评估、核心效果、时长、趋势、细分群体及诊断结果，给出唯一决策。
除非用户单独要求，否则不得启动、暂停、结束、调整流量、删除或推出实验。这些操作不属于解读范畴，需用户明确确认目标及影响后才可执行。

Output requirements

输出要求

  • Use the explicitly requested language, otherwise the language of the user's latest substantive message. Localize all user-facing prose, headings, labels, statuses, conclusions, warnings, limitations, and next actions; treat section names in this Skill as semantic guidance and remove unintended mixed-language output.
  • Keep code, commands, raw IDs, event/property/metric names, Feature keys, SDK/API names, and official enum values unchanged when translation would alter their technical meaning.
  • Lead with the decision status.
  • Use an adaptive report, not a fixed numbered template. Organize the default user-facing hierarchy as: conclusion, core impact, experiment decision, and product optimization or next action.
  • Keep data reliability, core metrics, and attention metrics inside the core impact section:
    • Data reliability validation: summarize the diagnostic checklist and state the overall result as reliable, limited, or unreliable. Keep normal checks compact and expand only warnings, failures, or unverified checks.
    • Core metrics: present the primary metric's control and treatment values, absolute and relative effect, interval and p-value, relation to MDE or the business threshold, and business meaning. Use causal wording only when the trust assessment supports it.
    • Attention metrics: present secondary and diagnostic metrics only when they affect the decision, explain the mechanism, or reveal material risk. Do not promote a post-hoc metric into the success criterion.
  • For a brief answer, collapse these subsections into compact paragraphs under core impact; do not promote them into unrelated top-level sections.
  • Treat labels such as 
    material blockers
    , 
    trust blockers
    , diagnostic severity names, and framework labels such as 
    Gate 1
    or 
    Gate 2
    as internal taxonomy. Never expose them as report headings or append their English forms after localized headings. Use natural user-facing wording such as “数据可靠性验证” or “当前为什么不能下结论” when those concepts need a heading.
  • Place metric trends and metric-related segments under the relevant core or attention metric. Add broader context, cause diagnosis, assumptions, or limitations only when they materially change interpretation.
  • Use a product-optimization section only when trustworthy evidence supports a specific product change. Otherwise use a next-validation or next-action section.
  • Include only content supported by the request and evidence. Do not add a section merely because it appeared in a previous report.
  • Separate platform observations, deterministic calculations, and inferred causes. State uncertainty and evidence gaps explicitly.
  • Do not return an unexplained raw table.
  • 使用用户明确要求的语言，否则使用用户最新实质性消息的语言。所有面向用户的文案、标题、标签、状态、结论、警告、限制及下一步操作均需本地化；将本Skill中的章节名称视为语义指引，避免出现非预期的混合语言输出。
  • 若翻译会改变技术含义，则代码、命令、原始ID、事件/属性/指标名称、功能键、SDK/API名称及官方枚举值保持不变。
  • 以决策状态开头。
  • 使用自适应报告，而非固定编号模板。默认面向用户的内容层级为：结论、核心影响、实验决策、产品优化或下一步操作。
  • 将数据可靠性、核心指标及重点指标放在核心影响部分内：
    • 数据可靠性验证：总结诊断检查清单，将整体结果表述为可靠、有限或不可靠。常规检查内容保持简洁，仅展开警告、失败或未验证的检查项。
    • 核心指标：展示核心指标的对照组与处理组取值、绝对及相对效果、区间与p值、与MDE或业务阈值的关系，以及业务意义。仅当可信度评估支持时，才可使用因果表述。
    • 重点指标：仅当次要及诊断指标影响决策、解释机制或暴露重大风险时，才予以展示。不得将事后选取的指标提升为成功标准。
  • 若需简短回答，可将这些子部分合并为核心影响下的紧凑段落；不得将其提升为无关的顶级章节。
  • 将
    material blockers
    、
    trust blockers
    、诊断严重程度名称及
    Gate 1
    或
    Gate 2
    等框架标签视为内部分类体系。不得将其作为报告标题，也不得在本地化标题后附加英文形式。当需要标题时，使用自然的面向用户的表述，例如“数据可靠性验证”或“当前无法下结论的原因”。
  • 将指标趋势及指标相关细分群体放在对应的核心或重点指标下。仅当更广泛的上下文、原因诊断、假设或限制会实质性改变解读时，才予以添加。
  • 仅当可信证据支持特定产品变更时，才设置产品优化章节；否则使用下一步验证或下一步操作章节。
  • 仅包含请求及证据支持的内容。不得因某章节出现在之前的报告中就添加该章节。
  • 区分平台观测结果、确定性计算结果及推论原因。明确说明不确定性及证据缺口。
  • 不得返回未加解释的原始表格。

Failure behavior

异常处理

  • Ambiguous project or experiment: show resolved candidates and ask; do not guess.
  • Missing report capability: preserve the 
    ae-cli
    gap and request a platform export with required fields.
  • Permission failure: stop dependent queries and report the missing permission.
  • Empty successful report: say no matching data; do not relabel it as a transport failure.
  • Partial success: use available evidence and enumerate failures.
  • Conflicting sources: prefer raw assignment/exposure evidence for data-quality diagnosis and explain the discrepancy.
  • 项目或实验不明确：展示已匹配的候选项并询问；不得猜测。
  • 缺少报告能力：保留
    ae-cli
    的功能缺口，请求包含所需字段的平台导出文件。
  • 权限失败：停止相关查询并报告缺失的权限。
  • 报告为空但执行成功：说明无匹配数据；不得将其标记为传输失败。
  • 部分成功：使用可用证据并列出失败项。
  • 数据源冲突：在数据质量诊断中优先使用原始分配/曝光证据，并解释差异。