grounded-rag-ingestion

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Grounded RAG Ingestion

Grounded RAG 数据摄入

Turn a document into a stream of provenance-carrying chunks — one JSONL line per chunk, each tagged with its element type, page index, bounding box, confidence score, and reading-order index, all traceable back to a specific region of the source page.

The differentiator is provenance, a capability frontier-LLM Markdown extraction structurally cannot supply: a downstream system can highlight the exact region of the source document a retrieval result came from. That is the compliance-grade requirement for RAG in finance, healthcare, and legal workflows where "the model said so" is not an acceptable citation. This skill does not relitigate raw Markdown quality (see the

pdf-to-markdown

skill for that) — it adds the provenance layer on top of Nutrient extraction.

This is a doer skill: a bundled

uv

-runnable Python script (

chunk.py

) drives the pipeline end to end and emits JSONL to stdout or a file. It is embedding-agnostic — it stops at the chunk boundary. No vector DB client, no embedding provider import.

将文档转换为带有溯源信息的分块流——每个分块对应JSONL的一行，均标记有元素类型、页码索引、边界框（bounding box）、置信度分数和阅读顺序索引，所有信息均可追溯回源文档的特定页面区域。

其核心优势在于溯源能力，这是前沿LLM的Markdown提取功能在结构上无法提供的：下游系统可以高亮显示检索结果源自源文档的精确区域。这是金融、医疗和法律工作流中RAG的合规级要求，在这些场景中“模型这么说的”并不算是可接受的引用。本技能不涉及原始Markdown质量的优化（相关需求请使用

pdf-to-markdown

技能）——它在Nutrient提取的基础上添加了溯源层。

这是一个执行类技能：一个打包好的可通过

uv

运行的Python脚本（

chunk.py

）端到端驱动流水线，并将JSONL输出到标准输出或文件中。它与嵌入方式无关——仅处理分块边界，不包含向量数据库客户端或嵌入提供商的导入。

When to use

使用场景

Grounded / cited RAG: retrieval results include
```
page_index
```
and
```
bbox
```
so the UI can draw a highlight box on the source document.
Confidence-aware ingestion: gate on
```
confidence >= threshold
```
to suppress low-confidence chunks before they reach the index.
Reading-order-aware chunking: preserve
```
reading_order
```
so sliding-window chunking respects the document's logical sequence.
Auditable pipelines: regulated workflows that must prove which page region a retrieved answer came from — the
```
bbox
```
+
```
page_index
```
tuple is the audit trail.

Not for cheap whole-document Markdown — that stays in

document-extraction-api

with

--output-format markdown

. Not for known target fields — if the request names specific fields ("the invoice number and total", "extract these fields", "map to my schema", "return these fields cited"), use

document-extraction-api

extract

(one call returns your schema's fields, each cited) instead of chunking the whole document for open-ended retrieval. Reach for this skill when retrieval is open-ended (semantic search over the whole document), not when the target fields are already known. Not for vector-store upsert or embedding generation — this skill stops at the chunk JSONL.

有依据/引用式RAG：检索结果包含
```
page_index
```
和
```
bbox
```
，以便UI可以在源文档上绘制高亮框。
置信度感知的数据摄入：通过
```
confidence >= 阈值
```
过滤低置信度分块，避免其进入索引。
阅读顺序感知的分块：保留
```
reading_order
```
，确保滑动窗口分块遵循文档的逻辑顺序。
可审计流水线：受监管的工作流必须证明检索答案源自文档的哪个页面区域——
```
bbox
```
+
```
page_index
```
组合就是审计线索。

不适用场景：低成本全文档Markdown生成——此类需求请使用带

--output-format markdown

参数的

document-extraction-api

。不适用已知目标字段的场景——如果请求指定了特定字段（如“发票编号和总额”、“提取这些字段”、“映射到我的schema”、“返回带引用的指定字段”），请使用

document-extraction-api

的

extract

接口（一次调用即可返回符合你schema的字段，且每个字段均带有引用），而非为开放式检索对全文档进行分块。当检索为开放式（对全文档进行语义搜索）而非目标字段已知时，才使用本技能。本技能不处理向量库插入或嵌入生成——仅生成分块JSONL。

Dependency:

document-extraction-api

依赖：

document-extraction-api

This skill calls

document-extraction-api

under the hood. Read that skill first for mode selection, key setup, and credit-cost guidance.

chunk.py

reuses its

create_client()

helper and

client.parse(..., output_format="spatial")

call — it does not re-implement the API call.

The extraction skill must be present in the same plugin at

../document-extraction-api/scripts/

. If it lives elsewhere, set

PARSE_SCRIPT_PATH

to its

scripts/

directory;

chunk.py

fails fast with a clear message if it cannot find it.

本技能在底层调用

document-extraction-api

。请先阅读该技能文档了解模式选择、密钥设置和信用成本说明。

chunk.py

复用了其

create_client()

辅助函数和

client.parse(..., output_format="spatial")

调用——并未重新实现API调用逻辑。

该提取技能必须位于同一插件的

../document-extraction-api/scripts/

目录下。如果位于其他位置，请设置

PARSE_SCRIPT_PATH

指向其

scripts/

chunk.py

会快速失败并给出清晰提示。

Setup

设置

DWS Extract is a separate product from DWS Processor and has its own API key.

Get a Nutrient DWS Extract API key at https://dashboard.nutrient.io/.

Export it as

NUTRIENT_EXTRACT_API_KEY

bash

export NUTRIENT_EXTRACT_API_KEY="pdf_live_..."

Run from the directory containing this SKILL.md:

bash

cd <directory containing this SKILL.md> && uv run scripts/chunk.py --help

If your tenant has migrated to global DWS API keys, a single key set as either

NUTRIENT_EXTRACT_API_KEY

NUTRIENT_API_KEY

works for both products.

DWS Extract是独立于DWS Processor的产品，拥有独立的API密钥。

在https://dashboard.nutrient.io/获取Nutrient DWS Extract API密钥。

将其导出为

NUTRIENT_EXTRACT_API_KEY

：

bash

export NUTRIENT_EXTRACT_API_KEY="pdf_live_..."

从包含本SKILL.md的目录运行：

bash

cd <包含本SKILL.md的目录> && uv run scripts/chunk.py --help

如果你的租户已迁移至全局DWS API密钥，设置为

NUTRIENT_EXTRACT_API_KEY

或

NUTRIENT_API_KEY

的单个密钥可用于两款产品。

Mode selection

模式选择

Mode selection for this skill follows the same rules as
document-extraction-api
(mode → cost → when to use). Pass

--mode

through to the parse call; the default is

structure

Mode	Cost (cr/pg)	When
`text`	1	Cheapest; incompatible with spatial — not usable here (this skill requires spatial)
`structure`	1.5	Default. OCR + spatial typed elements with bounds and tables
`understand`	9	AI layout: required for `keyValueRegion` and `formula` elements
`agentic`	18	VLM: adds `altDescription` on pictures

Mode-gating — important.
keyValueRegion
and
formula
elements are only populated by
understand
mode or higher. Under the default
structure
mode the key-value and formula chunk paths produce nothing.
chunk.py
warns on stderr when KV/formula content would be expected but the mode is too low. To exercise key-value chunking, pass
--mode understand
.

本技能的模式选择遵循与

document-extraction-api

相同的规则（模式→成本→适用场景）。将

--mode

参数传递给解析调用；默认模式为

structure

。

模式	成本（cr/页）	适用场景
`text`	1	成本最低；与空间输出不兼容——无法在此使用（本技能需要空间输出）
`structure`	1.5	默认值。OCR + 带边界和表格的空间类型元素
`understand`	9	AI布局：`keyValueRegion` 和 `formula` 元素所需
`agentic`	18	VLM：为图片添加 `altDescription`

模式限制——重要提示。
keyValueRegion
和
formula
元素仅在
understand
模式或更高版本中生成。在默认的
structure
模式下，键值对和公式分块路径将无输出。当预期会有键值对/公式内容但模式级别过低时，
chunk.py
会在标准错误输出中发出警告。要使用键值对分块功能，请传递
--mode understand
参数。

Invocation

调用方式

bash

uv run scripts/chunk.py --input doc.pdf --out chunks.jsonl \
  [--doc-id ID] [--mode structure] \
  [--strategy element|reading-order-window|table-row] [--window-size 512] \
  [--min-confidence 0.0] [--skip-pictures] [--yes]

```
--out -
```
writes JSONL to stdout; otherwise to the named file (created with
```
0600
```
permissions — the JSONL carries extracted document text).
Credit usage is printed to stderr after the parse call.

bash

uv run scripts/chunk.py --input doc.pdf --out chunks.jsonl \
  [--doc-id ID] [--mode structure] \
  [--strategy element|reading-order-window|table-row] [--window-size 512] \
  [--min-confidence 0.0] [--skip-pictures] [--yes]

```
--out -
```
将JSONL写入标准输出；否则写入指定文件（文件权限为
```
0600
```
——JSONL包含提取的文档文本）。
解析调用完成后，信用使用情况将打印至标准错误输出。

Provenance chunk schema

溯源分块 schema

Each JSONL line is one chunk:

Field	Type	Source
`chunk_id`	string	deterministic — see below
`doc_id`	string	`--doc-id` or content hash of input bytes (never the basename)
`source_doc`	string	input basename, display-only
`element_type`	string	`paragraph` , `table` , `table_row` , `key_value_pair` , `formula` , `picture` , `handwriting`
`page_index`	int\|null	`element.page.pageIndex` (null when the element carries no page — never fabricated `0` )
`reading_order`	int\|null	`element.readingOrder` (null sorts last)
`bbox`	object\|null	`{x, y, width, height}` (normalized at runtime — see OQ-1; null when bounds are missing, never `{0,0,0,0}` )
`confidence`	float\|null	element-level `confidence` (0–1); null when unknown
`text`	string	element text per type-dispatch rules
`chunking_warning`	string?	present only on table span-expansion fallback chunks

chunk_id
is deterministic:

{doc_id}__p{page_index}_r{reading_order}_e{element_index}{disc}

```
_e{element_index}
```
(the element's position in the
```
(page_index, reading_order)
```
sort) guarantees inter-element uniqueness even when
```
reading_order
```
is
```
null
```
or shared by two elements.
```
{disc}
```
is the intra-element discriminator for elements that emit multiple chunks:
```
_tr{row}
```
(table-row),
```
_kv{pair_index}
```
(key-value pairs),
```
_w{window_index}
```
(reading-order windows); empty for single-chunk elements.
```
doc_id
```
is
```
--doc-id
```
if supplied, else a short content hash of the input bytes — never the basename, so two same-named files in different directories get disjoint IDs and upsert stays idempotent and tenant-safe.

Full spec:

references/provenance-chunk-schema.md

每个JSONL行对应一个分块：

字段	类型	来源
`chunk_id`	string	确定性生成——详见下文
`doc_id`	string	`--doc-id` 参数或输入字节的内容哈希（绝不是文件名）
`source_doc`	string	输入文件名，仅用于显示
`element_type`	string	`paragraph` , `table` , `table_row` , `key_value_pair` , `formula` , `picture` , `handwriting`
`page_index`	int\|null	`element.page.pageIndex` （当元素无页码时为null——绝不会生成 `0` ）
`reading_order`	int\|null	`element.readingOrder` （为null时排在最后）
`bbox`	object\|null	`{x, y, width, height}` （运行时归一化——详见OQ-1；当边界缺失时为null，绝不会是 `{0,0,0,0}` ）
`confidence`	float\|null	元素级别的 `confidence` （0–1）；未知时为null
`text`	string	根据元素类型规则生成的元素文本
`chunking_warning`	string?	仅在表格跨列扩展回退分块时出现

chunk_id
为确定性生成：

{doc_id}__p{page_index}_r{reading_order}_e{element_index}{disc}

。

```
_e{element_index}
```
（元素在
```
(page_index, reading_order)
```
排序中的位置）确保即使
```
reading_order
```
为null或两个元素共享同一
```
reading_order
```
，元素间仍保持唯一性。
```
{disc}
```
是生成多个分块的元素的元素内区分符：
```
_tr{row}
```
（表格行）、
```
_kv{pair_index}
```
（键值对）、
```
_w{window_index}
```
（阅读顺序窗口）；单个分块的元素则为空。
```
doc_id
```
若提供
```
--doc-id
```
参数则使用该值，否则为输入字节的短内容哈希——绝不是文件名，因此不同目录下同名文件会获得不同ID，确保插入操作具有幂等性且租户安全。

完整规范：

references/provenance-chunk-schema.md

。

Chunking strategies

分块策略

```
--strategy element
```
(default): one chunk per typed spatial element — the purest expression of the grounding value prop. A
```
table
```
becomes one whole-table TSV chunk.
```
--strategy table-row
```
: each table row becomes its own chunk (
```
_tr{row}
```
) — better precision on wide financial tables.
```
--strategy reading-order-window
```
: sliding window over reading-order-sorted text elements (
```
--window-size
```
in tokens,
```
_w{window_index}
```
) — better retrieval coherence for prose, at some provenance precision (a window may span two elements' bboxes).

```
--strategy element
```
（默认）：每个类型化空间元素对应一个分块——最纯粹地体现了溯源价值。一个
```
table
```
会成为一个完整表格的TSV分块。
```
--strategy table-row
```
：每个表格行对应一个独立分块（
```
_tr{row}
```
）——在宽幅财务表格上精度更高。
```
--strategy reading-order-window
```
：对按阅读顺序排序的文本元素进行滑动窗口分块（
```
--window-size
```
为令牌数，
```
_w{window_index}
```
）——对散文类文档的检索连贯性更好，但溯源精度会有所降低（一个窗口可能跨越两个元素的bbox）。

Cost gate

成本限制

chunk.py

runs its own preflight before the network call: it counts the input PDF's pages locally and estimates

pages × mode-cost

. If the estimate exceeds 200 credits (or the API's

remainingCredits

when known), it refuses to proceed unless

--yes

is passed or interactive confirmation is given. (The extraction skill's 200-credit rule is an agent instruction, not enforceable code — this gate is real code here.) For non-PDF inputs where local page count is unavailable, the gate requires

--yes

for

understand

agentic

runs.

chunk.py

在网络调用前会执行独立的预检：本地统计输入PDF的页数并估算

页数 × 模式成本

。如果估算值超过200信用（或已知API的

remainingCredits

），除非传递

--yes

参数或进行交互式确认，否则将拒绝执行。（提取技能的200信用规则是代理指令，而非可强制执行的代码——此处的限制是真实代码实现的。）对于无法本地统计页数的非PDF输入，

understand

agentic

模式运行时需要

--yes

参数。

Illustrative: passing chunks to an embedder

示例：将分块传递给嵌入模型

The boundary is the chunk schema. See

examples/embed-chunks-illustrative.py

for a minimal, illustrative-only example of loading the JSONL and passing

text

to an embedder while keeping

chunk_id

bbox

page_index

provenance attached to each embedding record. Swap the provider import for your stack — no embedding library is a dependency of this skill.

边界为分块schema。请查看

examples/embed-chunks-illustrative.py

，这是一个极简的仅作示例的代码片段，展示如何加载JSONL并将

text

传递给嵌入模型，同时将

chunk_id

bbox

page_index

溯源信息附加到每个嵌入记录中。可根据你的技术栈替换提供商导入——本技能不依赖任何嵌入库。

Anti-patterns

反模式

Do not use this skill when raw Markdown is sufficient — use
```
document-extraction-api
```
with
```
--output-format markdown
```
.
Do not bundle vector-DB or embedding-provider logic into
```
chunk.py
```
— the boundary is the chunk JSONL.
Do not expect key-value or formula chunks under the default
```
structure
```
mode — they require
```
--mode understand
```
.
Do not derive
```
doc_id
```
from the filename basename — it breaks cross-document collision safety (R15).

当原始Markdown已满足需求时，请勿使用本技能——请使用带
```
--output-format markdown
```
参数的
```
document-extraction-api
```
。
请勿将向量数据库或嵌入提供商逻辑集成到
```
chunk.py
```
中——边界为分块JSONL。
请勿期望在默认
```
structure
```
模式下生成键值对或公式分块——这些需要
```
--mode understand
```
参数。
请勿从文件名派生
```
doc_id
```
——这会破坏跨文档冲突安全性（R15）。

Security Hardening Addendum

安全加固补充说明

Never store
```
NUTRIENT_EXTRACT_API_KEY
```
in committed files. Use process env injection at runtime (shell/export, secrets manager, or host env).
The output JSONL carries extracted document text (potentially finance/health/legal content) and is written with
```
0600
```
permissions. Treat it as sensitive; do not commit it.
```
source_doc
```
carries the input basename, which may encode PII (e.g.
```
patient-jsmith.pdf
```
). For sensitive corpora, supply a sanitized
```
--doc-id
```
and be deliberate about what reaches the index.
The script prints credit usage (the numeric cost only) to stderr and never logs the API key.

切勿将
```
NUTRIENT_EXTRACT_API_KEY
```
存储在已提交的文件中。请在运行时通过进程环境注入（shell/export、密钥管理器或主机环境）。
输出的JSONL包含提取的文档文本（可能包含金融/医疗/法律内容），并以
```
0600
```
权限写入。请将其视为敏感数据；请勿提交到版本控制系统。
```
source_doc
```
包含输入文件名，可能包含PII（例如
```
patient-jsmith.pdf
```
）。对于敏感语料库，请提供经过清理的
```
--doc-id
```
，并谨慎控制哪些信息进入索引。
脚本仅将信用使用情况（仅数字成本）打印到标准错误输出，绝不会记录API密钥。

Rules

规则

Always require spatial output — Markdown output loses provenance and is rejected.
Preserve the printed credit-usage summary so the operator can observe per-call cost.
Reuse
```
document-extraction-api
```
's
```
create_client()
```
and
```
client.parse()
```
; do not re-implement the API call or mode-selection logic.

始终要求空间输出——Markdown输出会丢失溯源信息，将被拒绝。
保留打印的信用使用摘要，以便操作人员查看每次调用的成本。
复用
```
document-extraction-api
```
的
```
create_client()
```
和
```
client.parse()
```
；请勿重新实现API调用或模式选择逻辑。

Reference map

参考映射

```
references/provenance-chunk-schema.md
```
— authoritative chunk schema,
```
chunk_id
```
derivation, chunking strategies, table span-expansion, key-value mapping.
```
examples/embed-chunks-illustrative.py
```
— illustrative embedding snippet (not a supported script).
Sibling
```
document-extraction-api/SKILL.md
```
— mode selection, credit costs, key setup.

```
references/provenance-chunk-schema.md
```
——权威分块schema、
```
chunk_id
```
生成规则、分块策略、表格跨列扩展、键值对映射。
```
examples/embed-chunks-illustrative.py
```
——嵌入示例代码片段（非受支持脚本）。
同级目录的
```
document-extraction-api/SKILL.md
```
——模式选择、信用成本、密钥设置。

grounded-rag-ingestion

Original

Translation

Grounded RAG Ingestion

Grounded RAG 数据摄入

When to use

使用场景

Dependency:
`document-extraction-api`

依赖：
`document-extraction-api`

Setup

设置

Mode selection

模式选择

Invocation

调用方式

Provenance chunk schema

溯源分块 schema

Chunking strategies

分块策略

Cost gate

成本限制

Illustrative: passing chunks to an embedder

示例：将分块传递给嵌入模型

Anti-patterns

反模式

Security Hardening Addendum

安全加固补充说明

Rules

规则

Reference map

参考映射

grounded-rag-ingestion

Original

Translation

Grounded RAG Ingestion

Grounded RAG 数据摄入

When to use

使用场景

Dependency: document-extraction-api

依赖：document-extraction-api

Setup

设置

Mode selection

模式选择

Invocation

调用方式

Provenance chunk schema

溯源分块 schema

Chunking strategies

分块策略

Cost gate

成本限制

Illustrative: passing chunks to an embedder

示例：将分块传递给嵌入模型

Anti-patterns

反模式

Security Hardening Addendum

安全加固补充说明

Rules

规则

Reference map

参考映射

Dependency:
`document-extraction-api`

依赖：
`document-extraction-api`