Bolt Pipeliner

Use this skill to turn a user's pipeline goal into a working, config-driven Bolt Pipeliner project. The framework is Python-first: jobs are modules exposing

process_data(self, input_tables)

, while

configs/etl_config.yaml

declares the layers, dependencies, storage, incremental policy, and data-quality tests.

The user's business requirements and naming preferences are important. The framework's project layout and runtime contracts are not optional. Resolve a conflict by explaining it and asking the user before changing the design.

Operating Rules

Inspect the repository before editing anything.
Determine whether the request targets a new project or an existing project.
Ask questions conversationally. Use the host agent's question or approval tool when available, but do not assume that a questionary terminal session is interactive.
Ask only questions that are not already answered by project files or the persisted environment profile.
Present the resolved environment and pipeline plan before making substantial changes.
Never read or print secret values from
```
.env
```
, shell configuration, or cloud credential files. Record only secret names or references to a secret manager.
Do not overwrite existing files, generated artifacts, or user changes without explicit approval.
Do not edit files under
```
_boltpipeliner/
```
; patch the upstream project instead.

Read the framework contract before creating or moving files. Read the questionnaire when collecting setup answers.

Workflow

1. Inspect the project

For an existing project, inspect at least:

```
git status --short
```
```
configs/bolt_environment.yaml
```
, if present
```
configs/etl_config.yaml
```
```
configs/spark/*.toml
```
, if present
```
pyproject.toml
```
,
```
requirements*.txt
```
, or the active environment metadata

etl/

macros/

models/

model_notebooks/

tests/

, and

outputs/

For a new project, confirm the target path and whether it is empty. Do not run

bolt init

against a non-empty directory because the scaffolder intentionally refuses to do so.

2. Interview the user

Collect the decisions described in

references/questionnaire.md

. At minimum, resolve the project goal, source data, engine, storage format, layers, runtime, locations, incremental policy, quality checks, orchestration, and optional ML requirements.

Ask for sample files, schemas, or precise transformation rules when the user wants implementation rather than a skeleton. Generate real transformations only when they are grounded in those inputs. Otherwise create a clear TODO scaffold and identify the missing information.

3. Confirm the environment

Show a concise summary containing:

project root and whether it is new or existing
engine and per-job base class choices
layer order and filesystem paths
input and output locations
catalog, schema, and incremental policy
execution environment, Spark profile, and orchestrator
requested tests and generated artifacts
dependency and vendoring choices

Wait for confirmation before modifying an existing project or generating business logic.

4. Scaffold or adapt

For a new project:

Choose the closest

bolt init

preset:

minimal

medallion

diamond

pandas

, or

polars

Use
```
--vendor
```
or
```
--no-vendor
```
according to the user's preference.
Run the command only after confirming the target is empty.
For custom layers or storage formats, start from the closest preset and then update the framework files deliberately.

For an existing project:

Treat the existing
```
etl_config.yaml
```
and directory tree as evidence, not as disposable scaffolding.
Preserve compatible conventions and make the smallest change that satisfies the request.
If
```
bolt_environment.yaml
```
is absent, derive what can be derived, ask for the missing decisions, and create the profile after confirmation.

After scaffolding or before adapting an existing project, re-read the files from disk. Do not rely only on the conversation or on assumptions about what a CLI preset wrote.

5. Implement the pipeline

Put every scheduled pipeline job under the declared
```
etl/<layer>/
```
path.
Give each job a matching
```
module
```
entry in
```
configs/etl_config.yaml
```
.
Implement the
```
process_data(self, input_tables)
```
contract and use the YAML aliases when reading inputs.
Put reusable, project-local transformations in
```
macros/
```
and import them from jobs.
Put project-specific data-quality checks in the job's
```
tests:
```
block and broader unit tests under
```
tests/
```
.
Put production ML code in
```
models/
```
and ML experimentation in
```
model_notebooks/
```
.
Keep generated documentation, DAGs, layer scripts, schema files, and ETL notebooks under
```
outputs/
```
; regenerate them instead of hand-editing them.
Use the engine-appropriate base class. Do not add Spark imports to Pandas or Polars jobs unless the user explicitly needs a mixed-engine project.

When source data or schemas are available, implement the transformations, partitions, incremental behavior, and checks. Run a small local validation when possible before claiming the pipeline is complete.

6. Generate and validate

From the project root, use the vendored shim when present, otherwise the installed

bolt

command:

bash

python bolt.py test --config configs/etl_config.yaml
python bolt.py generate documentation --config configs/etl_config.yaml
python bolt.py generate all --config configs/etl_config.yaml

Use

main.py

bolt run

for execution only after the configuration and imports validate. Run the bundled layout validator when available:

bash

python skills/bolt-pipeliner/scripts/validate_layout.py .

If the skill was installed into an agent directory, use the validator from the skill's installed path instead of assuming it exists in the project.

Do not claim a cloud run succeeded without the required credentials and runtime. Report skipped checks and unresolved placeholders explicitly.

Placement Contract

The following locations are mandatory unless the framework itself is changed:

Content	Location
Raw/file ingestion jobs	`etl/_flatfile/`
Bronze jobs	`etl/0_bronze/`
Silver jobs	`etl/1_silver/`
Gold jobs	`etl/2_gold/`
Diamond and ML jobs	`etl/3_diamond/`
Custom layer jobs	`etl/<custom-layer>/`
Shared transforms	`macros/`
Production model code	`models/`
ML notebooks	`model_notebooks/`
Generated ETL notebook	`outputs/notebook/`
Generated artifacts	`outputs/`
Runtime configuration	`configs/etl_config.yaml`
Spark profile configuration	`configs/spark/<profile>.toml`

The current framework does not use a generic top-level

notebooks/

directory. Do not invent one for generated ETL or ML notebooks. The ETL notebook generator is currently Spark-oriented; do not generate a misleading ETL notebook for a Pandas or Polars project unless the current generator has been made engine-aware. ML notebooks scaffolded by

bolt init

are separate and are engine-aware.

Preference Resolution

Honor the user's preferences for:

business names and descriptions
source formats and locations
engine and storage choices
layer names when they still describe the dependency order
scheduling and orchestration
partitioning and incremental windows
quality thresholds
documentation, notebook, Airflow, and vendoring choices

The following framework constraints take precedence over convenience:

```
etl_config.yaml
```
is the runtime source of truth.
Job modules must expose
```
process_data(self, input_tables)
```
.
Dependencies must be expressed through
```
input_tables
```
.
Reusable code belongs in
```
macros/
```
.
Generated output belongs under
```
outputs/
```
.
Secrets must stay in the environment or a secret manager.

bolt-pipeliner

NPX Install

Tags

SKILL.md Content