lora-qlora-recipes

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

LoRA & QLoRA Recipes

LoRA & QLoRA 实操配置方案

This skill assumes the routing decision already happened —

finetuning-method-selection

should have already pointed here because the data shape is demonstrations (SFT), not preference pairs or a verifiable reward signal. What follows is the current best-practice recipe for configuring the adapter itself: which modules to target, how to size rank and alpha, what learning rate to use, and when QLoRA buys real headroom versus when it just adds risk. Dataset preparation and quality checks are a separate concern — see

dataset-curation

Input: a routing decision (SFT via LoRA/ QLoRA) plus a target size class. Output format: a validated adapter config — the kwarg values below, not free-form advice — that

llm-finetuning-training-engineer

consumes directly when it generates a runnable script.

本技能假设路由决策已完成——

finetuning-method-selection

（微调方法选择）技能应已引导至此，因为数据类型为演示样本（SFT，监督微调），而非偏好对或可验证的奖励信号。以下是配置适配器本身的当前最佳实践方案：包括选择目标模块、设置rank和alpha的大小、确定学习率，以及QLoRA何时能真正提升内存空间、何时会带来风险。数据集准备与质量检查属于独立范畴——详见

dataset-curation

（数据集整理）技能。

输入：路由决策（通过LoRA/QLoRA进行SFT）+ 目标模型规模类别。 输出格式：经过验证的适配器配置——即下方的关键字参数值，而非自由形式建议——供

llm-finetuning-training-engineer

（大模型微调训练工程师）技能直接用于生成可运行脚本。

The Reference Recipe

参考方案

The reference recipe is "LoRA Without Regret" (Thinking Machines/Schulman, 2025-09), now the settled convention for LoRA/QLoRA SFT.

参考方案为《LoRA Without Regret》（Thinking Machines/Schulman，2025-09），现已成为LoRA/QLoRA SFT的通用标准。

Target Modules

目标模块

Target all-linear modules, not just attention:

python

target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",   # attention
    "gate_proj", "up_proj", "down_proj",      # MLP — matters most
]

The MLP layers (

gate_proj

up_proj

down_proj

) matter most — attention-only targeting was the older, weaker convention. Dropping modules to save memory is a Failure Mode below, not a valid optimization.

目标为所有线性模块，而非仅注意力模块：

python

target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",   # attention
    "gate_proj", "up_proj", "down_proj",      # MLP — matters most
]

MLP层（

gate_proj

、

up_proj

、

down_proj

）最为关键——仅针对注意力模块是旧版、效果较弱的做法。为节省内存而删减模块属于下文提到的失效模式，并非合理优化手段。

Alpha and Learning Rate

Alpha值与学习率

lora_alpha = 2 * r
is the settled convention (NeurIPS 2025 "intruder dimensions" result). Don't hand-tune alpha independently of rank — derive it from rank every time.
LoRA learning rate ≈ 10x the equivalent full-fine-tune LR. For QLoRA specifically, 2e-4 is the standard starting point. Full hyperparameter tables and worked examples:
```
references/hyperparameters.md
```
.

lora_alpha = 2 * r
是通用标准（NeurIPS 2025「侵入维度」研究成果）。不要独立于rank手动调整alpha——每次都应从rank值推导得出。
LoRA学习率 ≈ 等效全量微调LR的10倍。针对QLoRA，2e-4是标准起始值。完整超参数表及示例详见：
```
references/hyperparameters.md
```
。

Rank by Task

按任务设置Rank值

Rank is task-shaped, not a single global default:

Task	Rank
RL (GRPO/RLVR adapters)	1–32
General default	16–32
SFT at scale	up to ~256

Higher rank isn't automatically better — it raises capacity to memorize as fast as it raises capacity to generalize. Start at the row matching the task, and only move up a row if the lower rank measurably underfits on held-out eval, not as a default hedge.

Rank值需匹配任务类型，而非单一全局默认值：

任务	Rank值
RL（GRPO/RLVR适配器）	1–32
通用默认值	16–32
大规模SFT	最高约256

Rank值并非越高越好——它提升泛化能力的同时，也会加快模型的记忆速度。先从匹配任务的行开始设置，仅当低Rank值在保留验证集上出现明显欠拟合时才上调，不要默认选择高Rank值。

Effective Batch Size

有效批处理大小

Keep effective batch size under 32. This recipe was validated at that scale — pushing effective batch higher is an untested extrapolation, not a free throughput win.

保持有效批处理大小低于32。本方案在此规模下已验证有效——增大有效批处理大小属于未测试的外推做法，无法保证吞吐量提升。

Unsloth Defaults

Unsloth默认配置

Unsloth is the reference implementation this plugin assumes as the default fast path — except for messages-shaped conversational SFT with

assistant_only_loss=True

, where Unsloth 2026.7.x's compiled trainer has no messages-shaped path at all and the plain-TRL escape hatch (

references/unsloth-trl-mapping.md

) is the default for that combination, not a rare-regression fallback. Its out-of-the-box defaults, and why each one is set that way:

lora_dropout=0
— the optimized kernel path assumes zero dropout; setting a nonzero value forfeits the fused-kernel speedup.
bias="none"
— bias terms add adapter parameters for negligible quality gain at this rank range.
use_gradient_checkpointing="unsloth"
— Unsloth's checkpointing variant, not vanilla HF checkpointing; saves roughly 30% VRAM over no checkpointing.
optim="adamw_8bit"
— 8-bit AdamW cuts optimizer-state memory with negligible quality impact at LoRA/QLoRA adapter scale.
random_state
fixed — pins LoRA initialization for reproducibility across runs; treat it like any other seed, not a tunable.

These show up together on the

get_peft_model

call:

python

model = FastLanguageModel.get_peft_model(
    model,
    r=32,
    target_modules=target_modules,
    lora_alpha=64,               # 2 * r
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

Exact kwarg names and their plain-TRL/PEFT equivalents, plus a full worked config including

SFTConfig

references/unsloth-trl-mapping.md

and

references/hyperparameters.md

Unsloth是本插件默认采用的快速路径参考实现——但对于

assistant_only_loss=True

的对话式SFT（消息格式数据），Unsloth 2026.7.x版本的编译训练器完全不支持消息格式路径，此时默认使用纯TRL替代方案（

references/unsloth-trl-mapping.md

），而非罕见回归的 fallback 方案。以下是其开箱即用的默认配置及设置原因：

lora_dropout=0
——优化内核路径假设dropout为0；设置非零值会失去融合内核带来的加速效果。
bias="none"
——偏差项会增加适配器参数，但在此Rank范围内对模型质量提升微乎其微。
use_gradient_checkpointing="unsloth"
——Unsloth的梯度检查点变体，而非原生HF检查点；相比不使用检查点可节省约30%显存。
optim="adamw_8bit"
——8位AdamW可减少优化器状态内存占用，且在LoRA/QLoRA适配器规模下对模型质量影响可忽略。
固定
random_state
——固定LoRA初始化以保证多轮训练的可复现性；将其视为种子而非可调参数。

这些参数会一同出现在

get_peft_model

调用中：

python

model = FastLanguageModel.get_peft_model(
    model,
    r=32,
    target_modules=target_modules,
    lora_alpha=64,               # 2 * r
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

精确的关键字参数名称及其纯TRL/PEFT等效参数，以及包含

SFTConfig

的完整可行配置详见：

references/unsloth-trl-mapping.md

和

references/hyperparameters.md

。

LoRA vs QLoRA vs Full FT

LoRA vs QLoRA vs 全量微调

Situation	Default choice
Adapting behavior on demonstrations	LoRA
Base model doesn't fit in bf16 at target rank	QLoRA
Injecting dense new domain knowledge	Full FT (see `finetuning-method-selection` )
Unsure which one	LoRA — upgrade to QLoRA only if memory forces it

QLoRA = NF4-quantized frozen base weights + BF16 adapters. This is what makes a 65B-class model trainable on 48GB — the quantized base is the memory win, not the adapter itself.
Full fine-tuning is not a default. Reserve it for dense knowledge injection where the goal is changing what the model knows at the weight level, not adapting a behavior. For everything else in this skill's scope, LoRA or QLoRA is the starting assumption.
On DGX Spark, QLoRA can OOM before an equivalent bf16 LoRA run would, even though QLoRA's steady-state footprint is smaller — bitsandbytes dequantization buffers are transient CUDA-side allocations that spike during load. A QLoRA OOM is not proof the model doesn't fit; the
```
dgx-spark-ops
```
plugin's
```
spark-memory-thermal-ops
```
skill covers the full OOM remediation ladder (bf16 LoRA is the next thing to try, not a further QLoRA shrink).

场景	默认选择
基于演示样本调整模型行为	LoRA
目标Rank下基础模型无法以bf16格式加载	QLoRA
注入密集型新领域知识	全量微调（参见 `finetuning-method-selection` ）
不确定选择哪种方式	LoRA — 仅当内存不足时升级为QLoRA

QLoRA = NF4量化冻结基础权重 + BF16适配器。这使得65B级模型可在48GB显存设备上训练——内存节省来自量化后的基础模型，而非适配器本身。
全量微调并非默认选项。仅在需要注入密集型知识（目标是从权重层面改变模型认知）而非调整行为时使用。本技能覆盖的所有其他场景，默认采用LoRA或QLoRA。
在DGX Spark上，QLoRA可能比等效bf16 LoRA更早出现OOM，尽管QLoRA的稳态显存占用更小——bitsandbytes的反量化缓冲区是CUDA侧的瞬时分配内存，会在加载时峰值增长。QLoRA出现OOM并不代表模型无法适配；
```
dgx-spark-ops
```
插件的
```
spark-memory-thermal-ops
```
技能涵盖完整的OOM修复流程（下一步尝试bf16 LoRA，而非进一步缩小QLoRA规模）。

Failure Modes

失效模式

fp16 divergence on non-BF16 GPUs. Training in fp16 on hardware that doesn't have solid BF16 support is a known source of loss spikes and silent divergence. Force
```
bf16=True
```
wherever the hardware supports it; don't fall back to fp16 as if it were equivalent. Check hardware support before picking a dtype:
bash
```
python -c "import torch; print(torch.cuda.is_bf16_supported())"
```
Rank too high on a small dataset overfits. A rank picked for "SFT at scale" (up to ~256) on a dataset that doesn't have scale behind it memorizes rather than generalizes. Match rank to the Rank by Task table above, not to the largest number available.
Removing target modules to save memory costs quality for negligible savings. The adapter parameters on
```
gate_proj
```
/
```
up_proj
```
/
```
down_proj
```
are a small fraction of total model size — cutting them barely moves memory but measurably hurts quality. If memory is tight, move to QLoRA or reduce rank/batch/pack length before trimming target modules.

All three failure modes share a pattern: they look like a training-loop bug (loss spikes, plateaus, memorization) but are actually a config choice that contradicts the reference recipe above. Check configuration against this skill before debugging the training loop itself.

非BF16 GPU上的fp16发散。在不支持BF16的硬件上以fp16训练会导致损失突增和静默发散。只要硬件支持，就强制设置
```
bf16=True
```
；不要将fp16视为等效替代方案。选择数据类型前先检查硬件支持：
bash
```
python -c "import torch; print(torch.cuda.is_bf16_supported())"
```
小数据集上Rank值过高导致过拟合。为「大规模SFT」选择的Rank值（最高约256）若用于小规模数据集，模型会偏向记忆而非泛化。请根据上述「按任务设置Rank值」表格匹配Rank值，而非选择最大可用值。
为节省内存删减目标模块会牺牲模型质量，且内存收益微乎其微。
```
gate_proj
```
/
```
up_proj
```
/
```
down_proj
```
上的适配器参数仅占模型总规模的一小部分——删减它们几乎不会减少内存占用，但会显著降低模型质量。若内存紧张，应先切换到QLoRA或减小Rank/批处理/打包长度，再考虑删减目标模块。

上述三种失效模式有共同特征：看似是训练循环漏洞（损失突增、停滞、记忆化），实则是与参考方案相悖的配置选择。调试训练循环前，请先对照本技能检查配置。

References

参考资料

```
references/hyperparameters.md
```
— full rank/ alpha/LR tables by task type, rsLoRA notes, batch/packing interactions, and a complete worked Unsloth config block.
```
references/unsloth-trl-mapping.md
```
— every Unsloth kwarg mapped to its TRL/PEFT equivalent, current TRL API notes, and the escape-hatch rule for when to drop back to plain TRL.

Related skills:

finetuning-method-selection

routes here;

dataset-curation

covers the data side this skill doesn't;

llm-finetuning-training-engineer

is the downstream consumer of the config this skill produces.

```
references/hyperparameters.md
```
——按任务类型划分的完整rank/alpha/LR表、rsLoRA说明、批处理/打包交互说明，以及完整的Unsloth配置示例。
```
references/unsloth-trl-mapping.md
```
——每个Unsloth关键字参数对应的TRL/PEFT等效参数、当前TRL API说明，以及何时退回到纯TRL的规则。