tao-train-visual-changenet

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Visual ChangeNet

Visual ChangeNet is a TAO Toolkit model for visual inspection and defect detection. It supports two tasks:

Classify — Binary image classification using a siamese-style architecture with a shared backbone (C-RADIO ViT) and a learnable difference module. Compares image pairs to classify defects as PASS/NO_PASS.
Segment — Pixel-level change segmentation using a ViT-Large NVDINOv2 backbone. Compares before/after image pairs to produce a binary change mask.

The backbone weight (

c_radio_v2_vit_base_patch16_224

) is the

nvidia/C-RADIOv2-B

model from HuggingFace, distributed as

model.safetensors

(~393 MB). The TAO 7.0.0-rc container does not auto-fetch from HF URLs —

ptm_utils.load_pretrained_weights()

hands the

pretrained_backbone_path

value to

torch.load(path)

safetensors.torch.load_file(path)

directly. Passing an

https://huggingface.co/...

URL or a repo id produces

FileNotFoundError

and the run fails with

Execution status: FAIL

within a few seconds. Stage the file locally before launch:

bash

python3 -c "from huggingface_hub import hf_hub_download; import shutil; \
shutil.copy(hf_hub_download('nvidia/C-RADIOv2-B', 'model.safetensors'), '<workspace>/backbone/c_radio_v2_b.safetensors')"

Mount it into the container (

-v <workspace>/backbone/c_radio_v2_b.safetensors:/data/pretrained_models/C-RADIOv2_B.safetensors

) and set the spec

model.backbone.pretrained_backbone_path

to the container path.

HF_TOKEN

is only needed at staging time, not at training time.

Visual ChangeNet是一款用于视觉检测和缺陷检测的TAO Toolkit模型。它支持两项任务：

分类 — 采用带有共享骨干网络（C-RADIO ViT）和可学习差异模块的孪生式架构进行二值图像分类。通过对比图像对将缺陷分为PASS/NO_PASS。
分割 — 使用ViT-Large NVDINOv2骨干网络进行像素级变化分割。对比前后图像对以生成二值变化掩码。

骨干权重（

c_radio_v2_vit_base_patch16_224

）是来自HuggingFace的

nvidia/C-RADIOv2-B

模型，以

model.safetensors

（约393 MB）格式分发。TAO 7.0.0-rc容器不会自动从HF URL获取权重 —

ptm_utils.load_pretrained_weights()

会直接将

pretrained_backbone_path

的值传递给

torch.load(path)

safetensors.torch.load_file(path)

。传入

https://huggingface.co/...

URL或仓库ID会导致

FileNotFoundError

，运行会在几秒内失败并显示

Execution status: FAIL

。请在启动前将文件预存到本地：

bash

python3 -c "from huggingface_hub import hf_hub_download; import shutil; \
shutil.copy(hf_hub_download('nvidia/C-RADIOv2-B', 'model.safetensors'), '<workspace>/backbone/c_radio_v2_b.safetensors')"

将其挂载到容器中（

-v <workspace>/backbone/c_radio_v2_b.safetensors:/data/pretrained_models/C-RADIOv2_B.safetensors

），并将配置

model.backbone.pretrained_backbone_path

设置为容器内的路径。

HF_TOKEN

仅在预存文件时需要，训练时不需要。

Dataclass Schemas

数据类模式

Generated TAO Core schemas are packaged in

schemas/<action>.schema.json

, with

schemas/manifest.json

listing available actions. Each generated schema also emits

references/spec_template_<action>.yaml

from the schema top-level

default

field. AutoML enablement is declared at the model layer in

references/skill_info.yaml

via

automl_enabled

. Runnable AutoML still requires

schemas/train.schema.json

and

references/spec_template_train.yaml

to exist and parse. Use the packaged train schema for

automl_default_parameters

automl_disabled_parameters

, defaults, min/max bounds, enums, option weights, math conditions, dependencies, and popular parameters. Do not expect

~/tao-core

at runtime; maintainers regenerate schemas/templates before packaging the skill bank.

生成的TAO Core模式打包在

schemas/<action>.schema.json

中，

schemas/manifest.json

列出了可用的操作。每个生成的模式还会从模式顶层的

default

字段生成

references/spec_template_<action>.yaml

。AutoML支持在

references/skill_info.yaml

中的模型层通过

automl_enabled

声明。可运行的AutoML仍要求

schemas/train.schema.json

和

references/spec_template_train.yaml

存在且可解析。使用打包的训练模式来设置

automl_default_parameters

、

automl_disabled_parameters

、默认值、最小/最大边界、枚举、选项权重、数学条件、依赖关系以及常用参数。运行时不要依赖

~/tao-core

；维护人员在打包技能库前会重新生成模式/模板。

Train Action Policy

训练操作策略

This model is AutoML-enabled at the model layer. Before handling any train-stage request, read

references/skill_info.yaml

and resolve the run override from either an explicit

automl_policy

value or the user's workflow request. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as

automl_policy: off

for this run only; otherwise default to

auto

. When

automl_policy: auto

automl_enabled: true

, and both

schemas/train.schema.json

and

references/spec_template_train.yaml

are packaged, route the train action through

tao-skill-bank:tao-run-automl

by default with this model's

skill_dir

. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and

automl_policy

. Use direct model training only when

automl_policy: off

or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.

Non-train actions such as

evaluate

inference

export

, and deploy flows stay in this model skill. The per-run

automl_policy

override does not change model metadata.

For TAO Deploy TensorRT actions (

gen_trt_engine

, TensorRT

evaluate

, and TensorRT

inference

for classify and segment variants), read

references/tao-deploy-visual-changenet.md

first. Deploy spec templates live in this skill's

references/

folder with the

spec_template_deploy_*.yaml

prefix.

该模型在模型层支持AutoML。处理任何训练阶段请求前，请读取

references/skill_info.yaml

，并通过显式的

automl_policy

值或用户的工作流请求来解析运行覆盖配置。将"turn off AutoML"、"disable AutoML"、"no HPO"或"plain training"这类短语视为本次运行的

automl_policy: off

；否则默认设为

auto

。当

automl_policy: auto

、

automl_enabled: true

且

schemas/train.schema.json

和

references/spec_template_train.yaml

均已打包时，默认将训练操作通过

tao-skill-bank:tao-run-automl

路由，并传入该模型的

skill_dir

。保留数据集、配置、输出目录、GPU/平台设置、父检查点和

automl_policy

的工作流/应用覆盖配置。仅当

automl_policy: off

或打包的训练模式/模板缺失时，才使用直接模型训练；如果模式缺失，需报告该模型已启用AutoML但无法运行，直到生成模式为止。

非训练操作（如

evaluate

、

inference

、

export

以及部署流程）仍在该模型技能中执行。每次运行的

automl_policy

覆盖配置不会更改模型元数据。

对于TAO Deploy TensorRT操作（

gen_trt_engine

、TensorRT

evaluate

以及分类和分割变体的TensorRT

inference

），请先阅读

references/tao-deploy-visual-changenet.md

。部署配置模板位于该技能的

references/

文件夹中，前缀为

spec_template_deploy_*.yaml

。

Tasks

任务

Classify (default)

分类（默认）

Uses actions:

train

evaluate

inference

. Defaults template:

references/spec_template_train.yaml

使用操作：

train

、

evaluate

、

inference

。默认模板：

references/spec_template_train.yaml

。

Segment

分割

Uses actions:

segment_train

segment_evaluate

segment_inference

. Defaults template:

references/spec_template_segment.yaml

Segmentation requires compiling custom CUDA ops (

MultiScaleDeformableAttention

) on first run, which takes ~5 minutes. The ViT adapter backbone uses these for multi-scale feature extraction.

Dataset structure for segmentation differs from classify — uses paired directories (

A/

B/

list/

label/

) instead of CSV files. See

dataset.segment.root_dir

in the defaults.

使用操作：

segment_train

、

segment_evaluate

、

segment_inference

。默认模板：

references/spec_template_segment.yaml

。

分割任务首次运行时需要编译自定义CUDA算子（

MultiScaleDeformableAttention

），耗时约5分钟。ViT适配器骨干网络使用这些算子进行多尺度特征提取。

分割任务的数据集结构与分类不同 — 使用配对目录（

A/

、

B/

、

list/

、

label/

）而非CSV文件。请参见默认配置中的

dataset.segment.root_dir

。

Datasets, Spec Overrides, and Data Format

数据集、配置覆盖与数据格式

Visual ChangeNet has two task modes with different dataset types and data source structures. Classify uses a 4-column CSV (

input_path,golden_path,label,object_name

) plus an images directory; segment uses a paired directory structure (

A/

B/

list/

label/

) under a single

root_dir

. Data source overrides are mandatory for every action — the agent MUST construct data source paths and include them in

spec_overrides

See

references/dataset-and-specs.md

for the full per-action dataset requirement tables (classify and segment), every spec-override example (train, export, quantize, evaluate, inference, gen_trt_engine for both variants), the classify CSV format, evaluate/inference and segment input fields, lighting conventions, segment data layout, and the

input_map

multi-lighting configuration.

Visual ChangeNet有两种任务模式，对应不同的数据集类型和数据源结构。分类任务使用4列CSV（

input_path,golden_path,label,object_name

）加图像目录；分割任务在单个

root_dir

下使用配对目录结构（

A/

、

B/

、

list/

、

label/

）。每个操作都必须覆盖数据源配置 — 代理必须构建数据源路径并将其包含在

spec_overrides

中。

有关分类和分割任务的各操作数据集要求表、所有配置覆盖示例（训练、导出、量化、评估、推理、两种变体的

gen_trt_engine

）、分类CSV格式、评估/推理和分割输入字段、光照约定、分割数据布局以及

input_map

多光照配置，请参见

references/dataset-and-specs.md

。

Local Docker Invocation

本地Docker调用

Without the TAO SDK, resolve the TAO pyt image from

versions.yaml

and invoke

visual_changenet <action>

directly with

--shm-size=8g

and the backbone

.ckpt

mounted as a single file. See

references/local-docker-invocation.md

for the full

docker run

command, the shared-memory requirement, the backbone mount detail, and the checkpoint/results_dir command-line override pattern.

如果没有TAO SDK，请从

versions.yaml

中解析TAO pyt镜像，直接调用

visual_changenet <action>

，并设置

--shm-size=8g

，同时将骨干网络

.ckpt

作为单个文件挂载。完整的

docker run

命令、共享内存要求、骨干网络挂载细节以及检查点/results_dir命令行覆盖模式，请参见

references/local-docker-invocation.md

。

Parameters, Hardware, and Error Patterns

参数、硬件与错误模式

Key knobs include

train.validation_interval

(default 50, must be ≤ num_epochs),

train.checkpoint_interval

(default 200, must be ≤ num_epochs),

train.num_epochs

(default 100),

model.classify.eval_margin

(default 0.3, the primary precision/recall threshold), and

train.classify.cls_weight

(default [1.0, 10.0]). Minimum hardware is 1 GPU with 16GB+ VRAM; 8 GPUs (DDP) are recommended for production. GPU count is managed internally by TAO — do not set

gpu_spec_key

See

references/parameters-and-troubleshooting.md

for the full parameter reference, hardware guidance, and the complete error-pattern catalog (checkpoint not found, CSV format mismatch, image extension mismatch, OOM, low eval accuracy, the contrastive-loss assertion, non-convergence, the segment-only MultiScaleDeformableAttention build, Lightning epoch misconfiguration, PYTHONPATH/ModuleNotFoundError, and epoch defaults).

关键参数包括

train.validation_interval

（默认50，必须≤训练轮数）、

train.checkpoint_interval

（默认200，必须≤训练轮数）、

train.num_epochs

（默认100）、

model.classify.eval_margin

（默认0.3，主要的精确率/召回率阈值）和

train.classify.cls_weight

（默认[1.0, 10.0]）。最低硬件要求为1块显存≥16GB的GPU；生产环境推荐使用8块GPU（DDP）。GPU数量由TAO内部管理 — 请勿设置

gpu_spec_key

。

完整的参数参考、硬件指南以及完整的错误模式目录（检查点未找到、CSV格式不匹配、图像扩展名不匹配、内存不足、评估准确率低、对比损失断言、不收敛、分割任务专属的MultiScaleDeformableAttention构建、Lightning训练轮数配置错误、PYTHONPATH/ModuleNotFoundError、训练轮数默认值问题），请参见

references/parameters-and-troubleshooting.md

。

Spec Param / Parent Model Inference

配置参数 / 父模型推理

Model-specific inference mappings belong in this MD file, not in

config.json

. Generated runners should read this section and apply the mappings with SDK helpers before

create_job()

. This mirrors the old microservices

infer_params.py

flow.

Inference mappings from this model skill:

Action	Spec Field	Inference Function	Meaning
evaluate	`results_dir`	`output_dir`	current job results directory
inference	`results_dir`	`output_dir`	current job results directory
train	`results_dir`	`output_dir`	current job results directory
train	`train.resume_training_checkpoint_path`	`resume_model`	model file inferred from the current job results folder

For

parent_model

parent_model_folder

, pass the upstream train/export/AutoML child job id as

parent_job_id

. The SDK lists the parent result folder, filters checkpoint artifacts, and returns the selected model file or folder. Do not add these mappings back to

config.json

and do not patch generated runner scripts to guess checkpoint paths.

模型特定的推理映射应放在此MD文件中，而非

config.json

。生成的运行器应读取本节内容，并在

create_job()

前使用SDK助手应用这些映射。这与旧微服务的

infer_params.py

流程一致。

该模型技能的推理映射：

操作	配置字段	推理函数	含义
evaluate	`results_dir`	`output_dir`	当前作业结果目录
inference	`results_dir`	`output_dir`	当前作业结果目录
train	`results_dir`	`output_dir`	当前作业结果目录
train	`train.resume_training_checkpoint_path`	`resume_model`	从当前作业结果文件夹推断出的模型文件

对于

parent_model

或

parent_model_folder

，将上游训练/导出/AutoML子作业ID作为

parent_job_id

传入。SDK会列出父结果文件夹，过滤检查点工件，并返回选定的模型文件或文件夹。请勿将这些映射添加回

config.json

，也不要修改生成的运行器脚本以猜测检查点路径。

Deployment

部署

tao-deploy-visual-changenet — Visual ChangeNet deploy workflow for TensorRT engine generation, TensorRT evaluation, and TensorRT inference using TAO Deploy.

tao-deploy-visual-changenet — 基于TAO Deploy的Visual ChangeNet部署工作流，用于TensorRT引擎生成、TensorRT评估和TensorRT推理。