Back to Details

experiment-design-kit

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Experiment Design Kit Skill

实验设计工具包Skill

When to Use

适用场景

  • Translating raw ideas into testable hypotheses with clear success metrics.
  • Ensuring experiment briefs include guardrails, instrumentation, and rollout details.
  • Coaching pods on best practices for multi-variant or multi-surface tests.
  • 将初步想法转化为带有明确成功指标的可测试假设。
  • 确保实验简报包含防护规则、监控埋点和上线细节。
  • 指导团队掌握多变体或多触点测试的最佳实践。

Framework

框架

  1. Problem Framing – define user problem, business impact, and north-star metric.
  2. Hypothesis Structure – "If we do X for Y persona, we expect Z change" with assumptions.
  3. Measurement Plan – primary metric, guardrails, min detectable effect, power calc.
  4. Variant Strategy – control definition, variant catalog, targeting, and exclusion rules.
  5. Operational Plan – owners, timeline, dependencies, QA/rollback steps.
  1. 问题界定 – 明确用户问题、业务影响和核心指标。
  2. 假设结构 – 采用“如果我们为Y用户群体实施X，我们预期会产生Z变化”的格式，并附上前提假设。
  3. 衡量计划 – 确定核心指标、防护规则、最小可检测效果和统计功效计算。
  4. 变体策略 – 定义对照组、变体目录、目标受众和排除规则。
  5. 执行计划 – 明确负责人、时间线、依赖项、QA及回滚步骤。

Templates

模板

  • Experiment brief (context, hypothesis, design, metrics, launch checklist).
  • Guardrail register with thresholds + alerting rules.
  • Variant matrix for surfaces, messaging, and states.
  • GTM Agents Growth Backlog Board – capture idea → sizing → prioritization scoring (ICE/RICE) @puerto/README.md#183-212.
  • Weekly Experiment Packet – includes KPI guardrails, qualitative notes, and next bets for Marketing Director + Sales Director.
  • Rollback Playbook – pre-built checklist tied to lifecycle-mapping rip-cord procedures.
  • 实验简报（包含背景、假设、设计、指标、上线检查清单）。
  • 带有阈值和告警规则的防护规则注册表。
  • 针对不同触点、话术和场景的变体矩阵。
  • GTM Agents 增长待办事项看板 – 记录从想法提出→规模评估→优先级评分（ICE/RICE）的流程，详情见@puerto/README.md#183-212。
  • 每周实验报告 – 包含KPI防护规则、定性分析笔记，以及为营销总监和销售总监准备的后续测试建议。
  • 回滚操作手册 – 与生命周期映射紧急触发流程绑定的预构建检查清单。

Tips

小贴士

  • Pressure-test hypotheses with counter-metrics to avoid local optima.
  • Document data constraints early to avoid rework during build.
  • Pair with 
    guardrail-scorecard
    to ensure sign-off before launch.
  • Apply GTM Agents cadence: Monday backlog groom, Wednesday build review, Friday learnings sync.
  • Require KPI guardrails per stage (activation, engagement, monetization) before authorizing build.
  • If a test risks Sales velocity, include Sales Director in approval routing per GTM Agents governance.
  • 用反向指标对假设进行压力测试，避免陷入局部最优。
  • 尽早记录数据限制条件，避免在开发阶段返工。
  • 搭配
    guardrail-scorecard
    使用，确保上线前获得审批。
  • 遵循GTM Agents的工作节奏：周一梳理待办事项，周三进行开发评审，周五同步学习成果。
  • 在批准开发前，要求为每个阶段（激活、留存、变现）设置KPI防护规则。
  • 如果测试可能影响销售效率，需根据GTM Agents的治理流程将销售总监纳入审批环节。

GTM Agents Experiment Operating Model

GTM Agents 实验运营模型

  1. Backlog Intake – ideas flow from GTM pods; Growth Marketer tags theme, objective, expected impact.
  2. Prioritization – score with RICE + qualitative "strategic fit" modifier; surface top 3 bets weekly.
  3. Design & Instrumentation – reference Serena/Context7 to patch code + confirm documentation.
  4. Launch & Monitor – use guardrail-scorecard to watch leading indicators (churn, complaints, latency).
  5. Learning Loop – run Sequential Thinking retro; document hypothesis, result, decision, follow-up in backlog card.
  1. 待办事项收集 – 想法来自GTM团队；增长营销人员标记主题、目标和预期影响。
  2. 优先级排序 – 采用RICE评分法结合定性的“战略契合度”调整；每周推送排名前三的测试项目。
  3. 设计与埋点 – 参考Serena/Context7进行代码补丁并确认文档。
  4. 上线与监控 – 使用guardrail-scorecard监控领先指标（用户流失率、投诉量、延迟）。
  5. 学习闭环 – 开展Sequential Thinking回顾；在待办事项卡片中记录假设、结果、决策和后续行动。

KPI Guardrails (GTM Agents Reference)

KPI防护规则（GTM Agents参考标准）

  • Activation rate change must stay within ±3% of baseline for Tier-1 segments.
  • Revenue per visitor cannot drop more than 2% for more than 48h.
  • Support tickets tied to experiment variant must remain <5% of total volume.
  • 核心用户群体的激活率变化必须保持在基准值的±3%范围内。
  • 每访客收入下降幅度不得超过2%且持续时间不超过48小时。
  • 与实验变体相关的支持工单占比必须低于总工单量的5%。

Weekly Experiment Packet Outline

每周实验报告大纲

Week Ending: <Date>

1. Portfolio Snapshot – tests live, status, KPI trend (guardrail vs actual)
2. Key Wins – hypothesis, uplift, next action (ship, iterate, expand)
3. Guardrail Alerts – what tripped, mitigation taken (rollback? scope adjust?)
4. Pipeline Impact – SQLs, ARR influenced, notable customer anecdotes
5. Upcoming Launches – dependencies, owners, open questions
Share packet with Growth, Marketing Director, Sales Director, and RevOps to mirror GTM Agents's cross-functional communication rhythm.
Week Ending: <Date>

1. 项目组合概览 – 进行中的测试、状态、KPI趋势（防护规则 vs 实际数据）
2. 关键成果 – 假设内容、提升效果、后续行动（上线、迭代、拓展）
3. 防护规则告警 – 触发告警的原因、已采取的缓解措施（回滚？调整范围？）
4. 业务管线影响 – SQL数量、受影响的ARR、重要客户反馈
5. 即将上线的测试 – 依赖项、负责人、待解决问题
将报告分享给增长团队、营销总监、销售总监和营收运营团队，以遵循GTM Agents的跨职能沟通节奏。