Grounded RAG Ingestion

Turn a document into a stream of provenance-carrying chunks — one JSONL line per chunk, each tagged with its element type, page index, bounding box, confidence score, and reading-order index, all traceable back to a specific region of the source page.

The differentiator is provenance, a capability frontier-LLM Markdown extraction structurally cannot supply: a downstream system can highlight the exact region of the source document a retrieval result came from. That is the compliance-grade requirement for RAG in finance, healthcare, and legal workflows where "the model said so" is not an acceptable citation. This skill does not relitigate raw Markdown quality (see the

pdf-to-markdown

skill for that) — it adds the provenance layer on top of Nutrient extraction.

This is a doer skill: a bundled

uv

-runnable Python script (

chunk.py

) drives the pipeline end to end and emits JSONL to stdout or a file. It is embedding-agnostic — it stops at the chunk boundary. No vector DB client, no embedding provider import.

When to use

Grounded / cited RAG: retrieval results include
```
page_index
```
and
```
bbox
```
so the UI can draw a highlight box on the source document.
Confidence-aware ingestion: gate on
```
confidence >= threshold
```
to suppress low-confidence chunks before they reach the index.
Reading-order-aware chunking: preserve
```
reading_order
```
so sliding-window chunking respects the document's logical sequence.
Auditable pipelines: regulated workflows that must prove which page region a retrieved answer came from — the
```
bbox
```
+
```
page_index
```
tuple is the audit trail.

Not for cheap whole-document Markdown — that stays in

document-extraction-api

with

--output-format markdown

. Not for known target fields — if the request names specific fields ("the invoice number and total", "extract these fields", "map to my schema", "return these fields cited"), use

document-extraction-api

extract

(one call returns your schema's fields, each cited) instead of chunking the whole document for open-ended retrieval. Reach for this skill when retrieval is open-ended (semantic search over the whole document), not when the target fields are already known. Not for vector-store upsert or embedding generation — this skill stops at the chunk JSONL.

Dependency:

document-extraction-api

This skill calls

document-extraction-api

under the hood. Read that skill first for mode selection, key setup, and credit-cost guidance.

chunk.py

reuses its

create_client()

helper and

client.parse(..., output_format="spatial")

call — it does not re-implement the API call.

The extraction skill must be present in the same plugin at

../document-extraction-api/scripts/

. If it lives elsewhere, set

PARSE_SCRIPT_PATH

to its

scripts/

directory;

chunk.py

fails fast with a clear message if it cannot find it.

Setup

DWS Extract is a separate product from DWS Processor and has its own API key.

Get a Nutrient DWS Extract API key at https://dashboard.nutrient.io/.

Export it as

NUTRIENT_EXTRACT_API_KEY

bash

export NUTRIENT_EXTRACT_API_KEY="pdf_live_..."

Run from the directory containing this SKILL.md:

bash

cd <directory containing this SKILL.md> && uv run scripts/chunk.py --help

If your tenant has migrated to global DWS API keys, a single key set as either

NUTRIENT_EXTRACT_API_KEY

NUTRIENT_API_KEY

works for both products.

Mode selection

Mode selection for this skill follows the same rules as
document-extraction-api
(mode → cost → when to use). Pass

--mode

through to the parse call; the default is

structure

Mode	Cost (cr/pg)	When
`text`	1	Cheapest; incompatible with spatial — not usable here (this skill requires spatial)
`structure`	1.5	Default. OCR + spatial typed elements with bounds and tables
`understand`	9	AI layout: required for `keyValueRegion` and `formula` elements
`agentic`	18	VLM: adds `altDescription` on pictures

Mode-gating — important.
keyValueRegion
and
formula
elements are only populated by
understand
mode or higher. Under the default
structure
mode the key-value and formula chunk paths produce nothing.
chunk.py
warns on stderr when KV/formula content would be expected but the mode is too low. To exercise key-value chunking, pass
--mode understand
.

Invocation

bash

uv run scripts/chunk.py --input doc.pdf --out chunks.jsonl \
  [--doc-id ID] [--mode structure] \
  [--strategy element|reading-order-window|table-row] [--window-size 512] \
  [--min-confidence 0.0] [--skip-pictures] [--yes]

```
--out -
```
writes JSONL to stdout; otherwise to the named file (created with
```
0600
```
permissions — the JSONL carries extracted document text).
Credit usage is printed to stderr after the parse call.

Provenance chunk schema

Each JSONL line is one chunk:

Field	Type	Source
`chunk_id`	string	deterministic — see below
`doc_id`	string	`--doc-id` or content hash of input bytes (never the basename)
`source_doc`	string	input basename, display-only
`element_type`	string	`paragraph` , `table` , `table_row` , `key_value_pair` , `formula` , `picture` , `handwriting`
`page_index`	int\|null	`element.page.pageIndex` (null when the element carries no page — never fabricated `0` )
`reading_order`	int\|null	`element.readingOrder` (null sorts last)
`bbox`	object\|null	`{x, y, width, height}` (normalized at runtime — see OQ-1; null when bounds are missing, never `{0,0,0,0}` )
`confidence`	float\|null	element-level `confidence` (0–1); null when unknown
`text`	string	element text per type-dispatch rules
`chunking_warning`	string?	present only on table span-expansion fallback chunks

chunk_id
is deterministic:

{doc_id}__p{page_index}_r{reading_order}_e{element_index}{disc}

```
_e{element_index}
```
(the element's position in the
```
(page_index, reading_order)
```
sort) guarantees inter-element uniqueness even when
```
reading_order
```
is
```
null
```
or shared by two elements.
```
{disc}
```
is the intra-element discriminator for elements that emit multiple chunks:
```
_tr{row}
```
(table-row),
```
_kv{pair_index}
```
(key-value pairs),
```
_w{window_index}
```
(reading-order windows); empty for single-chunk elements.
```
doc_id
```
is
```
--doc-id
```
if supplied, else a short content hash of the input bytes — never the basename, so two same-named files in different directories get disjoint IDs and upsert stays idempotent and tenant-safe.

Full spec:

references/provenance-chunk-schema.md

Chunking strategies

```
--strategy element
```
(default): one chunk per typed spatial element — the purest expression of the grounding value prop. A
```
table
```
becomes one whole-table TSV chunk.
```
--strategy table-row
```
: each table row becomes its own chunk (
```
_tr{row}
```
) — better precision on wide financial tables.
```
--strategy reading-order-window
```
: sliding window over reading-order-sorted text elements (
```
--window-size
```
in tokens,
```
_w{window_index}
```
) — better retrieval coherence for prose, at some provenance precision (a window may span two elements' bboxes).

Cost gate

chunk.py

runs its own preflight before the network call: it counts the input PDF's pages locally and estimates

pages × mode-cost

. If the estimate exceeds 200 credits (or the API's

remainingCredits

when known), it refuses to proceed unless

--yes

is passed or interactive confirmation is given. (The extraction skill's 200-credit rule is an agent instruction, not enforceable code — this gate is real code here.) For non-PDF inputs where local page count is unavailable, the gate requires

--yes

for

understand

agentic

runs.

Illustrative: passing chunks to an embedder

The boundary is the chunk schema. See

examples/embed-chunks-illustrative.py

for a minimal, illustrative-only example of loading the JSONL and passing

text

to an embedder while keeping

chunk_id

bbox

page_index

provenance attached to each embedding record. Swap the provider import for your stack — no embedding library is a dependency of this skill.

Anti-patterns

Do not use this skill when raw Markdown is sufficient — use
```
document-extraction-api
```
with
```
--output-format markdown
```
.
Do not bundle vector-DB or embedding-provider logic into
```
chunk.py
```
— the boundary is the chunk JSONL.
Do not expect key-value or formula chunks under the default
```
structure
```
mode — they require
```
--mode understand
```
.
Do not derive
```
doc_id
```
from the filename basename — it breaks cross-document collision safety (R15).

Security Hardening Addendum

Never store
```
NUTRIENT_EXTRACT_API_KEY
```
in committed files. Use process env injection at runtime (shell/export, secrets manager, or host env).
The output JSONL carries extracted document text (potentially finance/health/legal content) and is written with
```
0600
```
permissions. Treat it as sensitive; do not commit it.
```
source_doc
```
carries the input basename, which may encode PII (e.g.
```
patient-jsmith.pdf
```
). For sensitive corpora, supply a sanitized
```
--doc-id
```
and be deliberate about what reaches the index.
The script prints credit usage (the numeric cost only) to stderr and never logs the API key.

Rules

Always require spatial output — Markdown output loses provenance and is rejected.
Preserve the printed credit-usage summary so the operator can observe per-call cost.
Reuse
```
document-extraction-api
```
's
```
create_client()
```
and
```
client.parse()
```
; do not re-implement the API call or mode-selection logic.

Reference map

```
references/provenance-chunk-schema.md
```
— authoritative chunk schema,
```
chunk_id
```
derivation, chunking strategies, table span-expansion, key-value mapping.
```
examples/embed-chunks-illustrative.py
```
— illustrative embedding snippet (not a supported script).
Sibling
```
document-extraction-api/SKILL.md
```
— mode selection, credit costs, key setup.

grounded-rag-ingestion

NPX Install

Tags

SKILL.md Content

Grounded RAG Ingestion

When to use

Dependency:
`document-extraction-api`

Setup

Mode selection

Invocation

Provenance chunk schema

Chunking strategies

Cost gate

Illustrative: passing chunks to an embedder

Anti-patterns

Security Hardening Addendum

Rules

Reference map