vision-sft

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Vision-Language SFT

This skill assumes

finetuning-method-selection

already routed here: the data shape is image+text demonstrations, not preference pairs or a verifiable reward signal, and the base is a vision-language model rather than a text-only one.

lora-qlora-recipes

covers the text-only LoRA/QLoRA recipe this skill specializes for the vision tower and projector; read that skill first if the LoRA fundamentals (rank, alpha, target modules) aren't already familiar.

Input: an image+text dataset and a VLM base model already picked from the model catalog. Output format: a validated adapter config — which components are frozen, LoRA target modules, and a

min_pixels

max_pixels

budget — that

llm-finetuning-training-engineer

consumes directly when it generates a runnable script.

本技能假设

finetuning-method-selection

已将任务路由至此：数据格式为图文演示数据，而非偏好对或可验证的奖励信号，基础模型为视觉语言模型而非纯文本模型。

lora-qlora-recipes

涵盖了纯文本场景下的LoRA/QLoRA方案，本技能则针对视觉塔和投影器进行了专门优化；如果您还不熟悉LoRA的基础概念（秩、阿尔法值、目标模块），请先阅读该技能文档。

输入： 已从模型目录中选定的图文数据集和VLM基础模型。 输出格式： 经过验证的适配器配置——包括哪些组件被冻结、LoRA目标模块，以及

min_pixels

max_pixels

预算——

llm-finetuning-training-engineer

可直接使用该配置生成可运行脚本。

Quick Reference

速查参考

Situation	Default
Adapting behavior on familiar images	Frozen tower+projector, LoRA r=8–16, α=16–32
Visual domain shift	Unfreeze last-6 ViT layers, vision LR 5–10x lower
Doesn't fit in bf16 at target rank	QLoRA — frozen vision tower only
`fast_inference=True`	`finetune_vision_layers=False`
Loss normal, eval not improving	Check the Two Silent Killers below first

场景	默认配置
在熟悉的图像上适配模型行为	冻结视觉塔+投影器，LoRA r=8–16，α=16–32
视觉领域迁移	解冻ViT最后6层，视觉学习率比LLM低5–10倍
目标秩下无法以bf16格式运行	使用QLoRA——仅冻结视觉塔
`fast_inference=True`	`finetune_vision_layers=False`
损失正常但评估指标无提升	先检查下文的两大“无声杀手”问题

The Consensus Recipe

通用方案

Freeze the vision tower and the projector. Put LoRA on the LLM only, all-linear (the same attention + MLP target list as text-only SFT — see

lora-qlora-recipes

), at r=8–16, α=16–32. This is the settled default for adapting a VLM's behavior without disturbing how it sees.

The vision tower and projector stay frozen by default. They already encode a general visual representation; retraining them is rarely necessary and adds risk without adding capability for most tasks.
LoRA rank runs lower than the text-only general default (r=8–16 here vs r=16–32 for text-only SFT) because the LLM-only adapter is adapting behavior, not injecting new visual knowledge.
QLoRA is permitted only with a frozen vision tower. Quantizing the base while also unfreezing and training vision layers is unsupported and unstable — treat this as a hard pairing rule, not a tunable. If the vision tower needs to unfreeze, drop QLoRA and use bf16 LoRA instead.

python

undefined

冻结视觉塔和投影器。仅在LLM上部署LoRA，覆盖所有线性层（目标模块列表与纯文本SFT相同——详见

lora-qlora-recipes

），配置为r=8–16，α=16–32。这是在不改变模型视觉感知能力的前提下，适配VLM行为的成熟默认方案。

视觉塔和投影器默认保持冻结状态。它们已编码通用视觉表示，重新训练它们很少有必要，且对大多数任务而言，只会增加风险而不会提升能力。
LoRA秩设置低于纯文本场景的通用默认值（此处r=8–16，而纯文本SFT为r=16–32），因为仅针对LLM的适配器是在适配行为，而非注入新的视觉知识。
仅当视觉塔冻结时才允许使用QLoRA。在解冻并训练视觉层的同时对基础模型进行量化是不被支持且不稳定的——这是硬性规则，而非可调参数。如果需要解冻视觉塔，请放弃QLoRA，改用bf16格式的LoRA。

python

undefined

freeze tower + projector; LoRA on LLM only

for name, param in model.named_parameters(): if "vision_tower" in name or "projector" in name: param.requires_grad = False

target_modules = [ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ] # LLM-only, all-linear — r=8-16, alpha=16-32

undefined

for name, param in model.named_parameters(): if "vision_tower" in name or "projector" in name: param.requires_grad = False

target_modules = [ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ] # LLM-only, all-linear — r=8-16, alpha=16-32

undefined

When to Unfreeze

何时解冻视觉层

Unfreezing vision layers is a deliberate escalation, not a default decision — reach for it only when the domain shift is visual, not textual.

Unfreeze only for visual domain shift. If the task is teaching new behavior on images the tower already understands (charts, everyday photos), the frozen-tower recipe above is sufficient. Unfreeze when the visual domain itself is unfamiliar to the tower — satellite imagery, medical scans, dense technical diagrams — and the frozen-tower recipe plateaus.
Last-6 ViT layers is the sweet spot. Unfreezing the final six vision-transformer layers (not the whole tower) measured +1.7pt DocVQA at ~1.75x training cost over the frozen baseline. Treat six layers as the ceiling worth paying for; going further spends compute without a matched result.
Vision LR must run 5–10x lower than the LLM LR when unfrozen. The vision tower's pretrained representation is more fragile than the LLM's adapter; the same LR for both risks overwriting the visual representation faster than the LLM adapter can compensate.
High LoRA rank on the patch- embedding layer risks NaN. If patch embedding is in the unfrozen set, keep its rank low and watch early-step loss closely — one of the most fragile places to apply LoRA in a VLM.

解冻视觉层是一种刻意的进阶操作，而非默认选择——仅当领域迁移是视觉层面而非文本层面时，才考虑使用该操作。

仅针对视觉领域迁移解冻。如果任务是在视觉塔已熟悉的图像（如图表、日常照片）上教授新行为，上述冻结视觉塔的方案已足够。当视觉塔对目标视觉领域不熟悉时（如卫星图像、医学扫描图、密集技术图表），且冻结视觉塔的方案已达到性能瓶颈，才需要解冻视觉层。
解冻ViT最后6层是最优选择。解冻视觉Transformer的最后6层（而非整个视觉塔），相较于冻结基线，在DocVQA任务上实现了**+1.7分的提升，训练成本约为1.75倍**。将6层视为值得投入计算资源的上限；进一步解冻更多层只会消耗计算资源，而无法获得匹配的性能提升。
解冻视觉层时，视觉学习率必须比LLM学习率低5–10倍。视觉塔的预训练表示比LLM的适配器更脆弱，如果两者使用相同的学习率，可能会导致视觉表示被覆盖的速度快于LLM适配器的适应速度。
在 patch-embedding 层使用高LoRA秩可能导致NaN值。如果patch embedding属于解冻集合，请将其秩设置为较低值，并密切关注训练初期的损失——这是VLM中应用LoRA最脆弱的位置之一。

The Two Silent Killers

两大“无声杀手”

Both produce a run that trains without error and without learning: the loss curve looks normal, the model doesn't improve, and neither throws an exception — both need an explicit pre-training check, not just a clean training log.

Image-tag/count mismatch. Every image placeholder token in the templated text must map 1:1 to a media item actually passed to the collator. A mismatch (one placeholder, zero or two images attached; or an image with no placeholder) doesn't error in most collators — it silently misaligns image and text, and the model "trains but learns nothing." Validate the 1:1 placeholder-to-media mapping before training starts, on every example, not just a sample. Full validation-checklist detail:
```
references/collators-and-pitfalls.md
```
.
min_pixels
/
max_pixels
resolution budget. This pair is the single most consequential hyperparameter for quality and memory in VLM SFT — more than rank, alpha, or LR. Too low silently downsamples images below what the task needs (small document text becomes unreadable even though training "succeeds"); too high blows the activation memory budget or forces too small a batch to train stably. Set it deliberately per dataset, don't leave it at a framework default.

这两种情况都会导致训练过程无报错但无学习效果：损失曲线看起来正常，但模型性能没有提升，且不会抛出任何异常——两者都需要明确的预训练检查，而不仅仅是查看训练日志是否干净。

图像与标签/数量不匹配。模板文本中的每个图像占位符标记必须与传递给整理器的媒体项保持1:1映射。不匹配（一个占位符对应零个或两个图像；或有图像但无占位符）在大多数整理器中不会报错——它会静默地导致图文错位，模型“看似在训练但实际上什么也没学到”。在训练开始前，必须验证每个示例的占位符与媒体项的1:1映射，而非仅抽样检查。详细的验证清单请参考：
```
references/collators-and-pitfalls.md
```
。
min_pixels
/
max_pixels
分辨率预算。这对参数是VLM SFT中影响模型质量和内存占用的最重要超参数——其影响超过秩、阿尔法值或学习率。设置过低会静默地将图像下采样到任务所需的最低分辨率以下（即使训练“成功”，文档中的小文本也会变得难以辨认）；设置过高则会超出激活内存预算，或迫使批量大小过小而无法稳定训练。需根据数据集刻意设置该参数，不要保留框架默认值。

Unsloth Specifics

Unsloth 专属细节

UnslothVisionDataCollator
is the collator Unsloth expects for VLM SFT — it handles the image-tag alignment and per-architecture processor contract described in
```
references/collators-and-pitfalls.md
```
. Don't substitute a text-only collator for VLM data.
finetune_vision_layers=False
is required when
fast_inference=True
. vLLM cannot serve LoRA adapters on vision layers, so a fast- inference setup that also unfreezes vision layers fails at serve time even if training succeeds. If the recipe calls for unfreezing the last-6 ViT layers (see When to Unfreeze above), fast inference is off the table for that run — choose one or the other, not both.

**
```
UnslothVisionDataCollator
```
**是Unsloth针对VLM SFT所需的整理器——它处理图像标签对齐和各架构处理器约定，详情请见
```
references/collators-and-pitfalls.md
```
。请勿用纯文本整理器替代VLM数据整理器。
当
fast_inference=True
时，必须设置
finetune_vision_layers=False
。vLLM无法在视觉层上部署LoRA适配器，因此即使训练成功，解冻视觉层的快速推理设置也会在部署时失败。如果方案要求解冻ViT最后6层（详见上文“何时解冻视觉层”），则该任务无法使用快速推理——两者只能选其一，不可同时启用。

Model Choice

模型选择

Base VLM choice is out of scope for this skill — it lives in one place, the model catalog at

finetuning-method-selection

references/model-catalog.md

. This skill and its references describe recipes by architecture family only, never by recommending one model over another.

VLM reinforcement learning (VLM-GRPO) is reference-only in this plugin — the fragmented tooling and reward-hacking failure modes specific to VLM-RL are covered in

grpo-rlvr-training

, not here. This skill's scope stops at supervised fine-tuning.

基础VLM的选择不在本技能的范围内——该功能统一由

finetuning-method-selection

的

references/model-catalog.md

模型目录管理。本技能及其参考文档仅按架构家族描述方案，不会推荐特定模型。

VLM强化学习（VLM-GRPO）在本插件中仅作参考——VLM-RL特有的工具碎片化和奖励黑客失效模式在

grpo-rlvr-training

中介绍，不在本技能范围内。本技能的范围仅限于监督微调。

Failure Modes

失效模式

The recurring mistake across every section above is treating a clean loss curve as proof the run is healthy. A normal-looking curve is consistent with both a working run and either silent killer, since the model trains on something either way — just not the aligned image-text signal when a killer is present. A flat eval score next to a normal loss curve means re-run the checklist in

references/collators-and-pitfalls.md

before touching any hyperparameter.

上文各部分反复提到的误区是，将平滑的损失曲线视为训练健康的证明。正常的损失曲线既可能对应正常运行的任务，也可能对应存在“无声杀手”问题的任务，因为无论哪种情况，模型都在“训练”——只是当存在“无声杀手”时，模型学习的并非对齐的图文信号。如果评估分数持平但损失曲线正常，在调整任何超参数之前，请重新运行

references/collators-and-pitfalls.md

中的检查清单。

References

参考资料

```
references/collators-and-pitfalls.md
```
— per- architecture collator table, dataset-format examples with image placeholders, a pre- training validation checklist, and the two- stage projector-alignment recipe as an advanced pattern.

Related skills:

finetuning-method-selection

routes here;

lora-qlora-recipes

covers the text-only LoRA fundamentals this skill specializes;

grpo-rlvr-training

covers VLM-RL (reference-only);

dataset-curation

covers image+text dataset preparation this skill doesn't.

```
references/collators-and-pitfalls.md
```
——按架构分类的整理器表格、带图像占位符的数据集格式示例、预训练验证清单，以及作为进阶模式的两阶段投影器对齐方案。