Kernel Debugging

内核调试

Purpose

用途

Guide agents through debugging the Linux kernel: kgdb over serial or USB, kdb built-in debugger commands, ftrace and

trace-cmd

, kprobes and kretprobes, dynamic debug (

dyndbg

), crash dump analysis with crash and makedumpfile, dmesg log levels, and

/proc/sys

runtime tuning for debugging.

指导开发者进行Linux内核调试：通过串口或USB使用kgdb、kdb内置调试器命令、ftrace与

trace-cmd

、kprobes与kretprobes、动态调试（

dyndbg

）、使用crash和makedumpfile分析崩溃转储、dmesg日志级别，以及用于调试的

/proc/sys

运行时调优。

When to Use

适用场景

Kernel panic or oops with unclear stack trace
Live debugging a kernel module or driver with kgdb
Tracing function calls without recompiling (ftrace, kprobes)
Enabling verbose driver logging with dyndbg
Analyzing production crash dumps from kdump
Tuning kernel runtime parameters for debug sessions

内核panic或oops且堆栈跟踪不清晰
使用kgdb对内核模块或驱动进行在线调试
无需重新编译即可追踪函数调用（ftrace、kprobes）
使用dyndbg启用详细的驱动日志
分析来自kdump的生产环境崩溃转储
为调试会话调优内核运行时参数

Workflow

工作流程

1. dmesg and log levels

1. dmesg与日志级别

bash

undefined

bash

undefined

Kernel ring buffer

dmesg -T -l err,warn dmesg -w # follow live

Log level (0=emerg .. 7=debug)

cat /proc/sys/kernel/printk

current default minimum boot-default

4 4 1 7

Temporarily increase verbosity

echo 8 > /proc/sys/kernel/printk


Kernel `printk` levels: `KERN_EMERG` through `KERN_DEBUG`. Driver `dev_dbg()` requires `dyndbg` or `DEBUG` define.

echo 8 > /proc/sys/kernel/printk


内核`printk`级别：`KERN_EMERG`至`KERN_DEBUG`。驱动的`dev_dbg()`需要`dyndbg`或`DEBUG`宏定义。

2. kgdb over serial

2. 串口kgdb

Kernel config:

CONFIG_KGDB

,

CONFIG_KGDB_SERIAL_CONSOLE

.

bash

undefined

内核配置：

CONFIG_KGDB

、

CONFIG_KGDB_SERIAL_CONSOLE

。

bash

undefined

Boot parameter (ttyS0, 115200)

kgdboc=ttyS0,115200 kgdbwait

Or at runtime

echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc echo g > /proc/sysrq-trigger # break into kgdb


GDB host side:

```bash

echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc echo g > /proc/sysrq-trigger # break into kgdb


GDB主机端：

```bash

Connect to remote serial

gdb vmlinux (gdb) set serial baud 115200 (gdb) target remote /dev/ttyUSB0 (gdb) bt (gdb) list *panic+0x20


USB variant: `CONFIG_KGDB_KDB` with `kgdboc=usb` on supported hardware.

gdb vmlinux (gdb) set serial baud 115200 (gdb) target remote /dev/ttyUSB0 (gdb) bt (gdb) list *panic+0x20


USB变体：在支持的硬件上，启用`CONFIG_KGDB_KDB`并设置`kgdboc=usb`。

3. kdb commands

3. kdb命令

When

CONFIG_KGDB_KDB

enabled, kdb provides in-kernel shell:

kdb commands (at SysRq break)
├── bt          — stack backtrace
├── lsmod       — loaded modules
├── md <addr>   — memory display
├── ps          — process list
├── id <cpu>    — CPU registers
├── go          — continue execution
└── cpu <n>     — switch CPU context

bash

undefined

启用

CONFIG_KGDB_KDB

后，kdb提供内核内shell：

kdb commands (at SysRq break)
├── bt          — stack backtrace
├── lsmod       — loaded modules
├── md <addr>   — memory display
├── ps          — process list
├── id <cpu>    — CPU registers
├── go          — continue execution
└── cpu <n>     — switch CPU context

bash

undefined

Enter kdb

echo k > /proc/sysrq-trigger

undefined

echo k > /proc/sysrq-trigger

undefined

4. ftrace

bash

undefined

bash

undefined

Enable function tracer

echo function > /sys/kernel/debug/tracing/current_tracer echo 1 > /sys/kernel/debug/tracing/tracing_on

Filter to specific function

echo schedule > /sys/kernel/debug/tracing/set_ftrace_filter echo ':probe_' >> /sys/kernel/debug/tracing/set_ftrace_notrace

Capture and read

cat /sys/kernel/debug/tracing/trace_pipe

Disable

echo 0 > /sys/kernel/debug/tracing/tracing_on echo nop > /sys/kernel/debug/tracing/current_tracer


```bash

echo 0 > /sys/kernel/debug/tracing/tracing_on echo nop > /sys/kernel/debug/tracing/current_tracer


```bash

trace-cmd (userspace recorder)

trace-cmd record -p function -l schedule,do_page_fault trace-cmd report trace-cmd.dat # kernelshark GUI

undefined

trace-cmd record -p function -l schedule,do_page_fault trace-cmd report trace-cmd.dat # kernelshark GUI

undefined

5. kprobes and kretprobes

5. kprobes与kretprobes

bash

undefined

bash

undefined

Dynamic kprobe via debugfs

echo 'p:myprobe do_sys_open %ax %si' > /sys/kernel/debug/tracing/kprobe_events echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable cat /sys/kernel/debug/tracing/trace

kretprobe

echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events

Cleanup

echo 0 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable echo '-:myprobe' > /sys/kernel/debug/tracing/kprobe_events


In-kernel kprobe (module):

```c
#include <linux/kprobes.h>

static struct kprobe kp = {
    .symbol_name = "do_sys_open",
    .pre_handler = my_pre_handler,
};

static int __init mod_init(void)
{
    return register_kprobe(&kp);
}

echo 0 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable echo '-:myprobe' > /sys/kernel/debug/tracing/kprobe_events


内核内kprobe（模块）：

```c
#include <linux/kprobes.h>

static struct kprobe kp = {
    .symbol_name = "do_sys_open",
    .pre_handler = my_pre_handler,
};

static int __init mod_init(void)
{
    return register_kprobe(&kp);
}

6. Dynamic debug (dyndbg)

6. 动态调试（dyndbg）

bash

undefined

bash

undefined

Enable all debug messages for a module

echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control

Enable specific file:line

echo 'file drivers/i2c/i2c-core.c +p' > /sys/kernel/debug/dynamic_debug/control

Query current settings

grep mydriver /sys/kernel/debug/dynamic_debug/control


Boot-time: `dyndbg="module mydriver +p"` on kernel command line.

grep mydriver /sys/kernel/debug/dynamic_debug/control


启动时配置：在内核命令行添加`dyndbg="module mydriver +p"`。

7. kdump and crash analysis

7. kdump与崩溃分析

bash

undefined

bash

undefined

Setup kdump (crash kernel reserved memory)

/etc/default/grub: crashkernel=256M

sudo kdump-config show

After crash, vmcore in /var/crash/

ls /var/crash/*/vmcore

Analyze with crash utility

crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore

crash> bt crash> log crash> ps crash> kmem -i crash> mod


```bash

crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore

crash> bt crash> log crash> ps crash> kmem -i crash> mod


```bash

Compress vmcore

makedumpfile -c -d /proc/vmcore /tmp/vmcore.lzo

undefined

makedumpfile -c -d /proc/vmcore /tmp/vmcore.lzo

undefined

8. /proc/sys runtime tuning for debugging

8. 用于调试的/proc/sys运行时调优

bash

undefined

bash

undefined

Panic on oops (reproduce in VM)

sysctl kernel.panic_on_oops=1

Soft lockup detection

sysctl kernel.softlockup_panic=1

Watchdog

sysctl kernel.nmi_watchdog=1

Slab debugging

Boot: slub_debug=P,pagealloc

undefined

undefined

9. Debugging decision tree

9. 调试决策树

Kernel issue type?
├── Panic/oops → dmesg; crash vmcore; CONFIG_DEBUG_INFO_BTF
├── Logic bug in driver → dyndbg; ftrace function_graph
├── Performance regression → perf record -g -a; trace-cmd
├── Intermittent → kprobes on suspect path
└── Module crash → kgdb; try_module_get audit

Kernel issue type?
├── Panic/oops → dmesg; crash vmcore; CONFIG_DEBUG_INFO_BTF
├── Logic bug in driver → dyndbg; ftrace function_graph
├── Performance regression → perf record -g -a; trace-cmd
├── Intermittent → kprobes on suspect path
└── Module crash → kgdb; try_module_get audit

Common Problems

常见问题

Symptom	Cause	Fix
kgdb won't connect	Wrong tty/baud	Match `kgdboc` to serial adapter
ftrace empty trace	tracing_off or nop tracer	`echo function > current_tracer` ; enable tracing_on
kprobe registration fails	Missing CONFIG_KPROBES or inlined function	Use tracepoint or static_key
dyndbg has no effect	CONFIG_DYNAMIC_DEBUG disabled	Rebuild kernel with option
crash can't read vmcore	Wrong vmlinux debug symbols	Install `linux-image-$(uname -r)-dbg`
SysRq doesn't work	`kernel.sysrq` disabled	`echo 1 > /proc/sys/kernel/sysrq`

症状	原因	解决方法
kgdb无法连接	错误的tty/波特率	将 `kgdboc` 与串口适配器匹配
ftrace追踪为空	tracing_off或nop追踪器	`echo function > current_tracer` ；启用tracing_on
kprobe注册失败	缺少CONFIG_KPROBES或函数被内联	使用tracepoint或static_key
dyndbg无效果	CONFIG_DYNAMIC_DEBUG未启用	重新编译内核并开启该选项
crash无法读取vmcore	错误的vmlinux调试符号	安装 `linux-image-$(uname -r)-dbg`
SysRq无法工作	`kernel.sysrq` 被禁用	`echo 1 > /proc/sys/kernel/sysrq`

kernel-debugging

Original

Translation

Kernel Debugging

内核调试

Purpose

用途

When to Use

适用场景

Workflow

工作流程

1. dmesg and log levels

1. dmesg与日志级别

Kernel ring buffer

Kernel ring buffer

Log level (0=emerg .. 7=debug)

Log level (0=emerg .. 7=debug)

current default minimum boot-default

current default minimum boot-default

4 4 1 7

4 4 1 7

Temporarily increase verbosity

Temporarily increase verbosity

2. kgdb over serial

2. 串口kgdb

Boot parameter (ttyS0, 115200)

Boot parameter (ttyS0, 115200)

Or at runtime

Or at runtime

Connect to remote serial

Connect to remote serial

3. kdb commands

3. kdb命令

Enter kdb

Enter kdb

4. ftrace

4. ftrace

Enable function tracer

Enable function tracer

Filter to specific function

Filter to specific function

Capture and read

Capture and read

Disable

Disable

trace-cmd (userspace recorder)

trace-cmd (userspace recorder)

5. kprobes and kretprobes

5. kprobes与kretprobes

Dynamic kprobe via debugfs

Dynamic kprobe via debugfs

kretprobe

kretprobe

Cleanup

Cleanup

6. Dynamic debug (dyndbg)

6. 动态调试（dyndbg）

Enable all debug messages for a module

Enable all debug messages for a module

Enable specific file:line

Enable specific file:line

Query current settings

Query current settings

7. kdump and crash analysis

7. kdump与崩溃分析

Setup kdump (crash kernel reserved memory)

Setup kdump (crash kernel reserved memory)

/etc/default/grub: crashkernel=256M

/etc/default/grub: crashkernel=256M

After crash, vmcore in /var/crash/

After crash, vmcore in /var/crash/

Analyze with crash utility

Analyze with crash utility

Compress vmcore

Compress vmcore

8. /proc/sys runtime tuning for debugging

8. 用于调试的/proc/sys运行时调优

Panic on oops (reproduce in VM)

Panic on oops (reproduce in VM)

Soft lockup detection