bolt-pipeliner

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Bolt Pipeliner

Use this skill to turn a user's pipeline goal into a working, config-driven Bolt Pipeliner project. The framework is Python-first: jobs are modules exposing

process_data(self, input_tables)

, while

configs/etl_config.yaml

declares the layers, dependencies, storage, incremental policy, and data-quality tests.

The user's business requirements and naming preferences are important. The framework's project layout and runtime contracts are not optional. Resolve a conflict by explaining it and asking the user before changing the design.

使用此技能可将用户的管道目标转化为可运行的、由配置驱动的Bolt Pipeliner项目。该框架以Python为核心：任务是暴露

process_data(self, input_tables)

方法的模块，而

configs/etl_config.yaml

则声明了层级、依赖关系、存储、增量策略以及数据质量测试。

用户的业务需求和命名偏好至关重要。框架的项目布局和运行时约定是不可更改的。若出现冲突，需先向用户说明冲突情况并征得同意后，再更改设计。

Operating Rules

操作规则

Inspect the repository before editing anything.
Determine whether the request targets a new project or an existing project.
Ask questions conversationally. Use the host agent's question or approval tool when available, but do not assume that a questionary terminal session is interactive.
Ask only questions that are not already answered by project files or the persisted environment profile.
Present the resolved environment and pipeline plan before making substantial changes.
Never read or print secret values from
```
.env
```
, shell configuration, or cloud credential files. Record only secret names or references to a secret manager.
Do not overwrite existing files, generated artifacts, or user changes without explicit approval.
Do not edit files under
```
_boltpipeliner/
```
; patch the upstream project instead.

Read the framework contract before creating or moving files. Read the questionnaire when collecting setup answers.

在编辑任何内容前，先检查仓库。
判断请求是针对新项目还是现有项目。
以对话形式提问。若有可用的主机Agent提问或审批工具，可使用该工具，但不要假设问卷终端会话是交互式的。
仅询问项目文件或持久化环境配置中未给出答案的问题。
在进行重大更改前，先展示已确定的环境和管道计划。
切勿读取或打印
```
.env
```
、Shell配置或云凭证文件中的机密值。仅记录机密名称或指向机密管理器的引用。
未经明确批准，不得覆盖现有文件、生成的工件或用户的更改。
不得编辑
```
_boltpipeliner/
```
下的文件；如需修改，应更新上游项目。

在创建或移动文件前，请阅读框架契约。收集设置答案时，请阅读问卷。

Workflow

工作流程

1. Inspect the project

1. 检查项目

For an existing project, inspect at least:

```
git status --short
```
```
configs/bolt_environment.yaml
```
, if present
```
configs/etl_config.yaml
```
```
configs/spark/*.toml
```
, if present
```
pyproject.toml
```
,
```
requirements*.txt
```
, or the active environment metadata

etl/

macros/

models/

model_notebooks/

tests/

, and

outputs/

For a new project, confirm the target path and whether it is empty. Do not run

bolt init

against a non-empty directory because the scaffolder intentionally refuses to do so.

对于现有项目，至少检查以下内容：

```
git status --short
```
```
configs/bolt_environment.yaml
```
（若存在）
```
configs/etl_config.yaml
```
```
configs/spark/*.toml
```
（若存在）
```
pyproject.toml
```
、
```
requirements*.txt
```
或活跃环境元数据

etl/

、

macros/

、

models/

、

model_notebooks/

、

tests/

和

outputs/

对于新项目，确认目标路径及其是否为空。不要在非空目录下运行

bolt init

，因为脚手架工具会主动拒绝此操作。

2. Interview the user

2. 与用户沟通

Collect the decisions described in

references/questionnaire.md

. At minimum, resolve the project goal, source data, engine, storage format, layers, runtime, locations, incremental policy, quality checks, orchestration, and optional ML requirements.

Ask for sample files, schemas, or precise transformation rules when the user wants implementation rather than a skeleton. Generate real transformations only when they are grounded in those inputs. Otherwise create a clear TODO scaffold and identify the missing information.

收集

references/questionnaire.md

中描述的决策信息。至少要确定项目目标、源数据、引擎、存储格式、层级、运行时、位置、增量策略、质量检查、编排以及可选的ML需求。

当用户需要实现而非框架搭建时，索要示例文件、模式或精确的转换规则。仅在有这些输入的基础上生成实际的转换逻辑；否则，创建清晰的TODO框架并指明缺失的信息。

3. Confirm the environment

3. 确认环境

Show a concise summary containing:

project root and whether it is new or existing
engine and per-job base class choices
layer order and filesystem paths
input and output locations
catalog, schema, and incremental policy
execution environment, Spark profile, and orchestrator
requested tests and generated artifacts
dependency and vendoring choices

Wait for confirmation before modifying an existing project or generating business logic.

展示包含以下内容的简洁摘要：

项目根目录及其是新项目还是现有项目
引擎和每个任务的基类选择
层级顺序和文件系统路径
输入和输出位置
目录、模式和增量策略
执行环境、Spark配置文件和编排器
请求的测试和生成的工件
依赖和供应商选择

在修改现有项目或生成业务逻辑前，等待用户确认。

4. Scaffold or adapt

4. 搭建或适配

For a new project:

Choose the closest

bolt init

preset:

minimal

medallion

diamond

pandas

, or

polars

Use
```
--vendor
```
or
```
--no-vendor
```
according to the user's preference.
Run the command only after confirming the target is empty.
For custom layers or storage formats, start from the closest preset and then update the framework files deliberately.

For an existing project:

Treat the existing
```
etl_config.yaml
```
and directory tree as evidence, not as disposable scaffolding.
Preserve compatible conventions and make the smallest change that satisfies the request.
If
```
bolt_environment.yaml
```
is absent, derive what can be derived, ask for the missing decisions, and create the profile after confirmation.

After scaffolding or before adapting an existing project, re-read the files from disk. Do not rely only on the conversation or on assumptions about what a CLI preset wrote.

对于新项目：

选择最接近的

bolt init

预设：

minimal

、

medallion

、

diamond

、

pandas

或

polars

。

根据用户偏好使用
```
--vendor
```
或
```
--no-vendor
```
参数。
仅在确认目标目录为空后运行命令。
对于自定义层级或存储格式，从最接近的预设开始，然后有意更新框架文件。

对于现有项目：

将现有的
```
etl_config.yaml
```
和目录结构视为依据，而非可随意丢弃的脚手架。
保留兼容的约定，做出满足请求的最小改动。
若
```
bolt_environment.yaml
```
不存在，推导可确定的信息，询问缺失的决策内容，确认后创建配置文件。

搭建完成后或适配现有项目前，重新从磁盘读取文件。不要仅依赖对话内容或对CLI预设生成内容的假设。

5. Implement the pipeline

5. 实现管道

Put every scheduled pipeline job under the declared
```
etl/<layer>/
```
path.
Give each job a matching
```
module
```
entry in
```
configs/etl_config.yaml
```
.
Implement the
```
process_data(self, input_tables)
```
contract and use the YAML aliases when reading inputs.
Put reusable, project-local transformations in
```
macros/
```
and import them from jobs.
Put project-specific data-quality checks in the job's
```
tests:
```
block and broader unit tests under
```
tests/
```
.
Put production ML code in
```
models/
```
and ML experimentation in
```
model_notebooks/
```
.
Keep generated documentation, DAGs, layer scripts, schema files, and ETL notebooks under
```
outputs/
```
; regenerate them instead of hand-editing them.
Use the engine-appropriate base class. Do not add Spark imports to Pandas or Polars jobs unless the user explicitly needs a mixed-engine project.

When source data or schemas are available, implement the transformations, partitions, incremental behavior, and checks. Run a small local validation when possible before claiming the pipeline is complete.

将所有计划的管道任务放在指定的
```
etl/<layer>/
```
路径下。
为每个任务在
```
configs/etl_config.yaml
```
中添加对应的
```
module
```
条目。
实现
```
process_data(self, input_tables)
```
契约，并在读取输入时使用YAML别名。
将可复用的、项目本地的转换逻辑放在
```
macros/
```
中，并在任务中导入。
将项目特定的数据质量检查放在任务的
```
tests:
```
块中，更通用的单元测试放在
```
tests/
```
下。
将生产级ML代码放在
```
models/
```
中，ML实验代码放在
```
model_notebooks/
```
中。
将生成的文档、DAG、层级脚本、模式文件和ETL笔记本放在
```
outputs/
```
下；如需修改，重新生成而非手动编辑。
使用适合引擎的基类。除非用户明确需要混合引擎项目，否则不要在Pandas或Polars任务中添加Spark导入。

当源数据或模式可用时，实现转换、分区、增量行为和检查。尽可能先运行一次本地小型验证，再宣称管道已完成。

6. Generate and validate

6. 生成和验证

From the project root, use the vendored shim when present, otherwise the installed

bolt

command:

bash

python bolt.py test --config configs/etl_config.yaml
python bolt.py generate documentation --config configs/etl_config.yaml
python bolt.py generate all --config configs/etl_config.yaml

Use

main.py

bolt run

for execution only after the configuration and imports validate. Run the bundled layout validator when available:

bash

python skills/bolt-pipeliner/scripts/validate_layout.py .

If the skill was installed into an agent directory, use the validator from the skill's installed path instead of assuming it exists in the project.

Do not claim a cloud run succeeded without the required credentials and runtime. Report skipped checks and unresolved placeholders explicitly.

从项目根目录出发，若存在供应商提供的垫片文件则使用该文件，否则使用已安装的

bolt

命令：

bash

python bolt.py test --config configs/etl_config.yaml
python bolt.py generate documentation --config configs/etl_config.yaml
python bolt.py generate all --config configs/etl_config.yaml

仅在配置和导入验证通过后，才使用

main.py

或

bolt run

执行任务。若有可用的捆绑布局验证工具，请运行：

bash

python skills/bolt-pipeliner/scripts/validate_layout.py .

若该技能已安装到Agent目录中，请使用技能安装路径下的验证工具，而非假设项目中存在该工具。

在没有所需凭证和运行时的情况下，不要宣称云运行成功。明确报告跳过的检查和未解决的占位符。

Placement Contract

放置约定

The following locations are mandatory unless the framework itself is changed:

Content	Location
Raw/file ingestion jobs	`etl/_flatfile/`
Bronze jobs	`etl/0_bronze/`
Silver jobs	`etl/1_silver/`
Gold jobs	`etl/2_gold/`
Diamond and ML jobs	`etl/3_diamond/`
Custom layer jobs	`etl/<custom-layer>/`
Shared transforms	`macros/`
Production model code	`models/`
ML notebooks	`model_notebooks/`
Generated ETL notebook	`outputs/notebook/`
Generated artifacts	`outputs/`
Runtime configuration	`configs/etl_config.yaml`
Spark profile configuration	`configs/spark/<profile>.toml`

The current framework does not use a generic top-level

notebooks/

directory. Do not invent one for generated ETL or ML notebooks. The ETL notebook generator is currently Spark-oriented; do not generate a misleading ETL notebook for a Pandas or Polars project unless the current generator has been made engine-aware. ML notebooks scaffolded by

bolt init

are separate and are engine-aware.

以下位置为强制要求，除非框架本身被修改：

内容	位置
原始/文件导入任务	`etl/_flatfile/`
Bronze任务	`etl/0_bronze/`
Silver任务	`etl/1_silver/`
Gold任务	`etl/2_gold/`
Diamond和ML任务	`etl/3_diamond/`
自定义层级任务	`etl/<custom-layer>/`
共享转换逻辑	`macros/`
生产级模型代码	`models/`
ML笔记本	`model_notebooks/`
生成的ETL笔记本	`outputs/notebook/`
生成的工件	`outputs/`
运行时配置	`configs/etl_config.yaml`
Spark配置文件	`configs/spark/<profile>.toml`

当前框架不使用通用的顶级

notebooks/

目录。请勿为生成的ETL或ML笔记本创建该目录。当前的ETL笔记本生成器以Spark为导向；除非生成器已支持多引擎，否则不要为Pandas或Polars项目生成易误导的ETL笔记本。

bolt init

搭建的ML笔记本是独立的，且支持多引擎。

Preference Resolution

偏好解决

Honor the user's preferences for:

business names and descriptions
source formats and locations
engine and storage choices
layer names when they still describe the dependency order
scheduling and orchestration
partitioning and incremental windows
quality thresholds
documentation, notebook, Airflow, and vendoring choices

The following framework constraints take precedence over convenience:

```
etl_config.yaml
```
is the runtime source of truth.
Job modules must expose
```
process_data(self, input_tables)
```
.
Dependencies must be expressed through
```
input_tables
```
.
Reusable code belongs in
```
macros/
```
.
Generated output belongs under
```
outputs/
```
.
Secrets must stay in the environment or a secret manager.

尊重用户的以下偏好：

业务名称和描述
源格式和位置
引擎和存储选择
层级名称（只要仍能描述依赖顺序）
调度和编排
分区和增量窗口
质量阈值
文档、笔记本、Airflow和供应商选择

以下框架约束优先于便利性：

```
etl_config.yaml
```
是运行时的唯一可信源。
任务模块必须暴露
```
process_data(self, input_tables)
```
方法。
依赖关系必须通过
```
input_tables
```
表达。
可复用代码应放在
```
macros/
```
中。
生成的输出应放在
```
outputs/
```
下。
机密必须保留在环境或机密管理器中。