Back to Details

systematic-debugging

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Systematic Debugging

系统化调试

Overview

概述

Random fixes waste time and create new bugs. Quick patches mask underlying issues.
Core principle: ALWAYS find root cause before attempting fixes. Symptom fixes are failure.
Violating the letter of this process is violating the spirit of debugging.
随意修复不仅浪费时间，还会引入新Bug。快速补丁只会掩盖潜在问题。
核心原则： 在尝试修复前，务必找到根本原因。仅修复症状等同于失败。
违反此流程的任何环节，都违背了调试的核心精神。

The Iron Law

铁律

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST
If you haven't completed Phase 1, you cannot propose fixes.
NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST
如果未完成第一阶段，不得提出修复方案。

When to Use

适用场景

Use for ANY technical issue:
  • Test failures
  • Bugs in production
  • Unexpected behavior
  • Performance problems
  • Build failures
  • Integration issues
Use this ESPECIALLY when:
  • Under time pressure (emergencies make guessing tempting)
  • "Just one quick fix" seems obvious
  • You've already tried multiple fixes
  • Previous fix didn't work
  • You don't fully understand the issue
Don't skip when:
  • Issue seems simple (simple bugs have root causes too)
  • You're in a hurry (rushing guarantees rework)
  • Manager wants it fixed NOW (systematic is faster than thrashing)
适用于所有技术问题：
  • 测试失败
  • 生产环境Bug
  • 异常行为
  • 性能问题
  • 构建失败
  • 集成问题
尤其适用于以下场景：
  • 处于时间压力下（紧急情况容易让人忍不住猜测）
  • “只需快速修复一下”看似显而易见
  • 已经尝试过多种修复方案
  • 之前的修复无效
  • 尚未完全理解问题
请勿跳过此流程的情况：
  • 问题看似简单（简单Bug也有根本原因）
  • 时间紧迫（仓促行事必然导致返工）
  • 经理要求立即修复（系统化调试比盲目尝试更快）

The Four Phases

四个阶段

You MUST complete each phase before proceeding to the next.
必须完成当前阶段后，才能进入下一阶段。

Phase 1: Root Cause Investigation

阶段1：根因调查

BEFORE attempting ANY fix:
  1. Read Error Messages Carefully
    • Don't skip past errors or warnings
    • They often contain the exact solution
    • Read stack traces completely
    • Note line numbers, file paths, error codes
  2. Reproduce Consistently
    • Can you trigger it reliably?
    • What are the exact steps?
    • Does it happen every time?
    • If not reproducible → gather more data, don't guess
  3. Check Recent Changes
    • What changed that could cause this?
    • Git diff, recent commits
    • New dependencies, config changes
    • Environmental differences
  4. Gather Evidence in Multi-Component Systems
    WHEN system has multiple components (CI → build → signing, API → service → database):
    BEFORE proposing fixes, add diagnostic instrumentation:
    For EACH component boundary:
  - Log what data enters component
  - Log what data exits component
  - Verify environment/config propagation
  - Check state at each layer

Run once to gather evidence showing WHERE it breaks
THEN analyze evidence to identify failing component
THEN investigate that specific component
    Example (multi-layer system):
    bash
    # Layer 1: Workflow
echo "=== Secrets available in workflow: ==="
echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

# Layer 2: Build script
echo "=== Env vars in build script: ==="
env | grep IDENTITY || echo "IDENTITY not in environment"

# Layer 3: Signing script
echo "=== Keychain state: ==="
security list-keychains
security find-identity -v

# Layer 4: Actual signing
codesign --sign "$IDENTITY" --verbose=4 "$APP"
    This reveals: Which layer fails (secrets → workflow ✓, workflow → build ✗)
  5. Trace Data Flow
    WHEN error is deep in call stack:
    See 
    root-cause-tracing.md
    in this directory for the complete backward tracing technique.
    Quick version:
    • Where does bad value originate?
    • What called this with bad value?
    • Keep tracing up until you find the source
    • Fix at source, not at symptom
在尝试任何修复之前：
  1. 仔细阅读错误信息
    • 不要跳过错误或警告
    • 它们通常包含确切的解决方案
    • 完整阅读堆栈跟踪（stack traces）
    • 记录行号、文件路径、错误代码
  2. 稳定复现问题
    • 能否可靠触发问题？
    • 确切步骤是什么？
    • 是否每次都会发生？
    • 如果无法复现 → 收集更多数据，不要猜测
  3. 检查近期变更
    • 哪些变更可能导致此问题？
    • Git diff、近期提交记录
    • 新依赖项、配置变更
    • 环境差异
  4. 在多组件系统中收集证据
    当系统包含多个组件时（CI → 构建 → 签名，API → 服务 → 数据库）：
    在提出修复方案前，添加诊断工具：
    For EACH component boundary:
  - Log what data enters component
  - Log what data exits component
  - Verify environment/config propagation
  - Check state at each layer

Run once to gather evidence showing WHERE it breaks
THEN analyze evidence to identify failing component
THEN investigate that specific component
    示例（多层系统）：
    bash
    # Layer 1: Workflow
echo "=== Secrets available in workflow: ==="
echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

# Layer 2: Build script
echo "=== Env vars in build script: ==="
env | grep IDENTITY || echo "IDENTITY not in environment"

# Layer 3: Signing script
echo "=== Keychain state: ==="
security list-keychains
security find-identity -v

# Layer 4: Actual signing
codesign --sign "$IDENTITY" --verbose=4 "$APP"
    这会揭示： 哪一层出现故障（密钥 → 工作流 ✓，工作流 → 构建 ✗）
  5. 追踪数据流
    当错误位于调用栈深处时：
    查看此目录下的 
    root-cause-tracing.md
    获取完整的反向追踪技巧。
    快速版：
    • 错误值源自何处？
    • 哪个调用传入了错误值？
    • 持续向上追踪直到找到根源
    • 在根源处修复，而非仅修复症状

Phase 2: Pattern Analysis

阶段2：模式分析

Find the pattern before fixing:
  1. Find Working Examples
    • Locate similar working code in same codebase
    • What works that's similar to what's broken?
  2. Compare Against References
    • If implementing pattern, read reference implementation COMPLETELY
    • Don't skim - read every line
    • Understand the pattern fully before applying
  3. Identify Differences
    • What's different between working and broken?
    • List every difference, however small
    • Don't assume "that can't matter"
  4. Understand Dependencies
    • What other components does this need?
    • What settings, config, environment?
    • What assumptions does it make?
修复前先找到模式：
  1. 寻找可正常运行的示例
    • 在同一代码库中定位类似的可正常运行代码
    • 哪些类似功能可以正常工作？
  2. 与参考实现对比
    • 如果是实现某种模式，请完整阅读参考实现
    • 不要略读 —— 逐行阅读
    • 在应用前完全理解该模式
  3. 识别差异
    • 正常运行的代码与故障代码之间有哪些差异？
    • 列出所有差异，无论多小
    • 不要假设“这无关紧要”
  4. 理解依赖关系
    • 该功能需要哪些其他组件？
    • 需要哪些设置、配置、环境？
    • 它有哪些隐含假设？

Phase 3: Hypothesis and Testing

阶段3：假设与测试

Scientific method:
  1. Form Single Hypothesis
    • State clearly: "I think X is the root cause because Y"
    • Write it down
    • Be specific, not vague
  2. Test Minimally
    • Make the SMALLEST possible change to test hypothesis
    • One variable at a time
    • Don't fix multiple things at once
  3. Verify Before Continuing
    • Did it work? Yes → Phase 4
    • Didn't work? Form NEW hypothesis
    • DON'T add more fixes on top
  4. When You Don't Know
    • Say "I don't understand X"
    • Don't pretend to know
    • Ask for help
    • Research more
遵循科学方法：
  1. 形成单一假设
    • 清晰陈述：“我认为X是根本原因，因为Y”
    • 将其记录下来
    • 要具体，不要模糊
  2. 最小化测试
    • 做出尽可能小的变更来验证假设
    • 一次只改变一个变量
    • 不要同时修复多个问题
  3. 验证后再继续
    • 有效？是 → 进入阶段4
    • 无效？形成新假设
    • 不要叠加更多修复
  4. 当你不确定时
    • 说出“我不理解X”
    • 不要假装懂
    • 请求帮助
    • 深入研究

Phase 4: Implementation

阶段4：实施修复

Fix the root cause, not the symptom:
  1. Create Failing Test Case
    • Simplest possible reproduction
    • Automated test if possible
    • One-off test script if no framework
    • MUST have before fixing
    • Use the 
      loom:test-driven-development
      skill for writing proper failing tests
  2. Implement Single Fix
    • Address the root cause identified
    • ONE change at a time
    • No "while I'm here" improvements
    • No bundled refactoring
  3. Verify Fix
    • Test passes now?
    • No other tests broken?
    • Issue actually resolved?
  4. If Fix Doesn't Work
    • STOP
    • Count: How many fixes have you tried?
    • If < 3: Return to Phase 1, re-analyze with new information
    • If ≥ 3: STOP and question the architecture (step 5 below)
    • DON'T attempt Fix #4 without architectural discussion
  5. If 3+ Fixes Failed: Question Architecture
    Pattern indicating architectural problem:
    • Each fix reveals new shared state/coupling/problem in different place
    • Fixes require "massive refactoring" to implement
    • Each fix creates new symptoms elsewhere
    STOP and question fundamentals:
    • Is this pattern fundamentally sound?
    • Are we "sticking with it through sheer inertia"?
    • Should we refactor architecture vs. continue fixing symptoms?
    Discuss with your human partner before attempting more fixes
    This is NOT a failed hypothesis - this is a wrong architecture.
修复根本原因，而非症状：
  1. 创建失败测试用例
    • 最简单的复现方式
    • 尽可能使用自动化测试
    • 如果没有测试框架，使用一次性测试脚本
    • 修复前必须完成此步骤
    • 可使用 
      loom:test-driven-development
      技能编写规范的失败测试用例
  2. 实施单一修复
    • 针对已确定的根本原因进行修复
    • 一次只做一项变更
    • 不要顺便进行“优化”
    • 不要捆绑重构
  3. 验证修复效果
    • 测试现在通过了吗？
    • 其他测试有没有被破坏？
    • 问题真的解决了吗？
  4. 如果修复无效
    • 停止操作
    • 统计：已经尝试了多少次修复？
    • 如果 <3：返回阶段1，结合新信息重新分析
    • 如果 ≥3：停止并质疑架构（见下文第5步）
    • 未经架构讨论，不得尝试第4次修复
  5. 如果3次以上修复失败：质疑架构
    表明存在架构问题的模式：
    • 每次修复都会在不同位置暴露出新的共享状态/耦合/问题
    • 修复需要“大规模重构”才能实现
    • 每次修复都会在其他地方引发新症状
    停止并质疑基础问题：
    • 这种模式从根本上合理吗？
    • 我们是不是“因惯性而坚持”？
    • 我们应该重构架构，还是继续修复症状？
    在尝试更多修复前，与你的人类伙伴讨论
    这不是假设失败 —— 而是架构存在问题。

Red Flags - STOP and Follow Process

危险信号 - 停止并遵循流程

If you catch yourself thinking:
  • "Quick fix for now, investigate later"
  • "Just try changing X and see if it works"
  • "Add multiple changes, run tests"
  • "Skip the test, I'll manually verify"
  • "It's probably X, let me fix that"
  • "I don't fully understand but this might work"
  • "Pattern says X but I'll adapt it differently"
  • "Here are the main problems: [lists fixes without investigation]"
  • Proposing solutions before tracing data flow
  • "One more fix attempt" (when already tried 2+)
  • Each fix reveals new problem in different place
ALL of these mean: STOP. Return to Phase 1.
If 3+ fixes failed: Question the architecture (see Phase 4.5)
如果你发现自己有以下想法：
  • “先快速修复，之后再调查”
  • “试试改X看看能不能行”
  • “同时做多项变更，然后运行测试”
  • “跳过测试，我手动验证”
  • “可能是X的问题，我来修复”
  • “我不完全理解，但这可能有用”
  • “模式要求X，但我要换种方式调整”
  • “主要问题有这些：[列出修复方案但未做调查]”
  • 在追踪数据流前就提出解决方案
  • “再试一次修复”（已经尝试2次以上）
  • 每次修复都会在不同地方暴露出新问题
以上所有情况都意味着：停止操作。返回阶段1。
如果3次以上修复失败： 质疑架构（见阶段4.5）

your human partner's Signals You're Doing It Wrong

你的人类伙伴认为你操作错误的信号

Watch for these redirections:
  • "Is that not happening?" - You assumed without verifying
  • "Will it show us...?" - You should have added evidence gathering
  • "Stop guessing" - You're proposing fixes without understanding
  • "Ultrathink this" - Question fundamentals, not just symptoms
  • "We're stuck?" (frustrated) - Your approach isn't working
When you see these: STOP. Return to Phase 1.
注意这些提醒：
  • “那不是真的吧？”——你未经验证就做出了假设
  • “能让我们看到...吗？”——你应该添加证据收集步骤
  • “别瞎猜”——你在未理解问题的情况下提出修复方案
  • “深入思考这个问题”——质疑基础问题，而非仅修复症状
  • “我们卡住了？”（语气沮丧）——你的方法行不通
当出现这些信号时：停止操作。返回阶段1。

Common Rationalizations

常见借口与真相

ExcuseReality
"Issue is simple, don't need process"Simple issues have root causes too. Process is fast for simple bugs.
"Emergency, no time for process"Systematic debugging is FASTER than guess-and-check thrashing.
"Just try this first, then investigate"First fix sets the pattern. Do it right from the start.
"I'll write test after confirming fix works"Untested fixes don't stick. Test first proves it.
"Multiple fixes at once saves time"Can't isolate what worked. Causes new bugs.
"Reference too long, I'll adapt the pattern"Partial understanding guarantees bugs. Read it completely.
"I see the problem, let me fix it"Seeing symptoms ≠ understanding root cause.
"One more fix attempt" (after 2+ failures)3+ failures = architectural problem. Question pattern, don't fix again.
借口真相
“问题很简单，不需要走流程”简单问题也有根本原因。此流程处理简单Bug速度很快。
“紧急情况，没时间走流程”系统化调试比盲目尝试更快。
“先试试这个，之后再调查”第一次修复会定下模式。从一开始就做对。
“确认修复有效后再写测试”未测试的修复无法持久。先写测试能验证问题。
“同时做多项修复节省时间”无法确定哪项有效。会引入新Bug。
“参考文档太长，我会调整模式”一知半解必然导致Bug。请完整阅读。
“我看到问题了，我来修复”看到症状≠理解根本原因。
“再试一次修复”（已经尝试2次以上）3次以上失败=架构问题。质疑模式，不要继续修复。

Quick Reference

快速参考

PhaseKey ActivitiesSuccess Criteria
1. Root CauseRead errors, reproduce, check changes, gather evidenceUnderstand WHAT and WHY
2. PatternFind working examples, compareIdentify differences
3. HypothesisForm theory, test minimallyConfirmed or new hypothesis
4. ImplementationCreate test, fix, verifyBug resolved, tests pass
阶段核心活动成功标准
1. 根因调查阅读错误信息、复现问题、检查变更、收集证据理解问题是什么及为什么发生
2. 模式分析寻找正常示例、对比参考实现识别差异
3. 假设验证形成理论、最小化测试假设得到确认或形成新假设
4. 实施修复创建测试用例、修复、验证Bug解决，测试通过

When Process Reveals "No Root Cause"

当流程显示“无根本原因”时

If systematic investigation reveals issue is truly environmental, timing-dependent, or external:
  1. You've completed the process
  2. Document what you investigated
  3. Implement appropriate handling (retry, timeout, error message)
  4. Add monitoring/logging for future investigation
But: 95% of "no root cause" cases are incomplete investigation.
如果系统化调查发现问题确实是环境、时间依赖或外部因素导致：
  1. 你已完成流程
  2. 记录调查内容
  3. 实施适当的处理逻辑（重试、超时、错误提示）
  4. 添加监控/日志以便未来调查
但注意： 95%的“无根本原因”案例都是调查不完整导致的。

Supporting Techniques

辅助技巧

These techniques are part of systematic debugging and available in this directory:
  • root-cause-tracing.md
     - Trace bugs backward through call stack to find original trigger
  • defense-in-depth.md
     - Add validation at multiple layers after finding root cause
  • condition-based-waiting.md
     - Replace arbitrary timeouts with condition polling
Related skills:
  • loom:test-driven-development - For creating failing test case (Phase 4, Step 1)
  • loom:verification-before-completion - Verify fix worked before claiming success
这些技巧属于系统化调试的一部分，可在此目录中找到：
  • root-cause-tracing.md
     —— 通过调用栈反向追踪Bug，找到最初的触发点
  • defense-in-depth.md
     —— 找到根本原因后，在多个层级添加验证逻辑
  • condition-based-waiting.md
     —— 用条件轮询替代任意超时
相关技能：
  • loom:test-driven-development —— 用于创建失败测试用例（阶段4，步骤1）
  • loom:verification-before-completion —— 在宣布成功前验证修复效果

Real-World Impact

实际效果

From debugging sessions:
  • Systematic approach: 15-30 minutes to fix
  • Random fixes approach: 2-3 hours of thrashing
  • First-time fix rate: 95% vs 40%
  • New bugs introduced: Near zero vs common
来自调试会话的数据：
  • 系统化方法：15-30分钟修复
  • 随意修复方法：2-3小时盲目尝试
  • 首次修复成功率：95% vs 40%
  • 引入新Bug数量：几乎为0 vs 常见

Journal Integration

日志集成

When operating on a task tracked under 
.agents/tasks/<task-id>/
, append a journal entry at this skill's milestone.
  • Trigger: after the root cause is identified or the investigation is paused
  • Reserved key(s): 
    note
    for findings; 
    blocker
    if a fix can't proceed
  • Entry shape: 
    ## <ISO8601-timestamp> — systematic-debugging
note: root cause identified — <one-line summary>
[optional ≤ 15-line body; longer content goes to artifacts/]
Resolve the task id from the explicit caller argument or 
.agents/tasks/.current
. If neither resolves, skip the append; do not guess.
See 
task-journal
for the full convention.
当处理 
.agents/tasks/<task-id>/
下跟踪的任务时，在此技能的里程碑处添加日志条目。
  • 触发时机： 确定根本原因后或调查暂停时
  • 保留关键字： 
    note
    用于记录发现；
    blocker
    用于记录无法继续修复的障碍
  • 条目格式： 
    ## <ISO8601-timestamp> — systematic-debugging
note: root cause identified — <单行摘要>
[可选≤15行内容；更长内容请放入artifacts/]
从显式调用参数或 
.agents/tasks/.current
解析任务ID。如果两者都无法解析，则跳过添加；不要猜测。
完整约定请参见 
task-journal
。