kernel-debugging-advanced

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Advanced Kernel Debugging

高级内核调试

Purpose

目的

Extend

skills/kernel/kernel-debugging

with production-grade tracing:

ftrace

trace-cmd

, kernel

perf

kprobes

kretprobes

kgdb

, crash dump analysis, and

printk

discipline. Not merged with

kernel-debugging

— that skill covers baseline kgdb/dyndbg; this one focuses on trace-cmd, function_graph, and live/post-mortem production workflows.

扩展

skills/kernel/kernel-debugging

，提供生产级追踪能力：

ftrace

、

trace-cmd

、内核

perf

、

kprobes

kretprobes

、

kgdb

、崩溃转储分析，以及

printk

使用规范。未合并到

kernel-debugging

——后者 skill 涵盖基础 kgdb/dyndbg；本 skill 专注于 trace-cmd、function_graph，以及生产环境中的实时/事后分析工作流。

When to Use

使用场景

Latency spikes in kernel driver without obvious bug
Tracing function call graph in kernel
Live inspection without recompiling (kprobes, bpf)
Post-mortem
```
vmcore
```
after panic

内核驱动程序出现延迟峰值，但没有明显的 bug
在内核中追踪函数调用图
无需重新编译即可进行实时检查（kprobes、bpf）
对 panic 后的
```
vmcore
```
进行事后分析

Workflow

工作流

1. ftrace function graph

bash

undefined

bash

undefined

Enable function graph tracer

echo function_graph > /sys/kernel/debug/tracing/current_tracer echo my_driver_probe > /sys/kernel/debug/tracing/set_graph_function echo 1 > /sys/kernel/debug/tracing/tracing_on

reproduce issue

cat /sys/kernel/debug/tracing/trace echo 0 > /sys/kernel/debug/tracing/tracing_on


Requires `CONFIG_FUNCTION_GRAPH_TRACER` and debugfs mounted.

cat /sys/kernel/debug/tracing/trace echo 0 > /sys/kernel/debug/tracing/tracing_on


需要启用 `CONFIG_FUNCTION_GRAPH_TRACER` 并挂载 debugfs。

2. trace-cmd record

bash

trace-cmd record -p function_graph -F my_probe
trace-cmd report
trace-cmd stat

Portable capture for sharing with others.

bash

trace-cmd record -p function_graph -F my_probe
trace-cmd report
trace-cmd stat

可移植的捕获结果，便于与他人共享。

3. perf in kernel context

3. 内核上下文中的 perf

bash

perf record -a -g -- sleep 10
perf report --stdio
perf probe --add my_driver:probe

See

skills/profilers/linux-perf

for userspace overlap.

bash

perf record -a -g -- sleep 10
perf report --stdio
perf probe --add my_driver:probe

与用户态重叠的部分，请参阅

skills/profilers/linux-perf

。

4. kprobes (dynamic)

4. kprobes（动态）

#include <linux/kprobes.h>

static struct kprobe kp = {
    .symbol_name = "do_sys_open",
    .pre_handler = handler,
};
register_kprobe(&kp);

Use sparingly in production; prefer static tracepoints when available.

#include <linux/kprobes.h>

static struct kprobe kp = {
    .symbol_name = "do_sys_open",
    .pre_handler = handler,
};
register_kprobe(&kp);

在生产环境中应谨慎使用；如果有可用的静态 tracepoint，应优先使用。

5. dyndbg and printk discipline

5. dyndbg 与 printk 使用规范

bash

echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control
echo 'file drivers/foo/*.c +p' > /sys/kernel/debug/dynamic_debug/control

pr_debug("state=%d\n", s);           /* compile-time optional */
printk_ratelimited(KERN_WARNING "hw fault\n");

bash

echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control
echo 'file drivers/foo/*.c +p' > /sys/kernel/debug/dynamic_debug/control

pr_debug("state=%d\n", s);           /* compile-time optional */
printk_ratelimited(KERN_WARNING "hw fault\n");

6. kgdb / kdb

bash

undefined

bash

undefined

kernel cmdline: kgdboc=ttyS0,115200 kgdbwait

echo g > /proc/sysrq-trigger # break into debugger (when configured)

undefined

echo g > /proc/sysrq-trigger # break into debugger (when configured)

undefined

7. Crash dump (vmcore)

7. 崩溃转储（vmcore）

bash

undefined

bash

undefined

after kdump capture

crash /usr/lib/debug/vmlinux vmcore crash> bt crash> dev -s

undefined

crash /usr/lib/debug/vmlinux vmcore crash> bt crash> dev -s

undefined

8. Agent usage

8. Agent 使用方式

/kernel-debugging-advanced Trace my_driver_ioctl latency with ftrace function_graph

/kernel-debugging-advanced Trace my_driver_ioctl latency with ftrace function_graph

Common Problems

常见问题

Symptom	Cause	Fix
Empty trace	Tracer not enabled	`tracing_on` ; check `current_tracer`
ftrace overhead	Graph all functions	Filter with `set_graph_function`
kprobe fail	Inlined symbol	Try nearby symbol or static key
kgdb no connect	Wrong `kgdboc` tty	Match serial/USB gadget
crash no symbols	Missing debuginfo	Install `linux-image-dbg` / vmlinux

症状	原因	修复方式
跟踪结果为空	未启用 Tracer	启用 `tracing_on` ；检查 `current_tracer`
ftrace 开销过高	跟踪了所有函数	使用 `set_graph_function` 进行过滤
kprobe 失败	符号被内联	尝试附近的符号或 static key
kgdb 无法连接	`kgdboc` tty 配置错误	匹配串行端口或 USB gadget
crash 中没有符号	缺少 debuginfo	安装 `linux-image-dbg` / vmlinux

kernel-debugging-advanced

Original

Translation

Advanced Kernel Debugging

高级内核调试

Purpose

目的

When to Use

使用场景

Workflow

工作流

1. ftrace function graph

1. ftrace function graph

Enable function graph tracer

Enable function graph tracer

reproduce issue

reproduce issue

2. trace-cmd record

2. trace-cmd record

3. perf in kernel context

3. 内核上下文中的 perf

4. kprobes (dynamic)

4. kprobes（动态）

5. dyndbg and printk discipline

5. dyndbg 与 printk 使用规范

6. kgdb / kdb

6. kgdb / kdb

kernel cmdline: kgdboc=ttyS0,115200 kgdbwait

kernel cmdline: kgdboc=ttyS0,115200 kgdbwait

7. Crash dump (vmcore)

7. 崩溃转储（vmcore）

after kdump capture

after kdump capture

8. Agent usage

8. Agent 使用方式

Common Problems

常见问题

Related Skills

相关 Skills