databricks-unstructured-pdf-generation

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Unstructured-Document for Demos and Eval Datasets on Databricks

在Databricks上用于演示和评估数据集的非结构化文档

Workflow for producing synthetic PDF documents + paired test questions as a Unity Catalog-resident dataset for Demos and RAG / unstructured-document retrieval evaluation on Databricks. The PDF-generation step uses standard local HTML → PDF tooling; the Databricks-specific value is the workflow shape — UC volume layout, paired question files, and integration with downstream Databricks retrieval /

ai_extract

ai_parse_document

evaluation.

本工作流用于生成合成PDF文档+配对测试问题，作为存储在Unity Catalog中的数据集，供Databricks上的演示以及RAG/非结构化文档检索评估使用。PDF生成步骤使用标准的本地HTML转PDF工具；Databricks专属价值体现在工作流形态——UC卷布局、配对问题文件，以及与下游Databricks检索/

ai_extract

ai_parse_document

评估的集成。

Workflow

工作流

Write HTML files to
```
./raw_data/html/
```
(write multiple files in parallel for speed) — domain-shaped to match the documents your retrieval pipeline will see in production.
Convert HTML → PDF using
```
<SKILL_ROOT>/scripts/pdf_generator.py
```
(parallel conversion, wraps
```
plutoprint
```
).
Upload PDFs to a Unity Catalog volume via
```
databricks fs cp
```
— same volume shape your production pipeline will read from.
Generate
```
./raw_data/pdf/pdf_eval_questions.json
```
pairing each document with retrieval-eval questions; this becomes the gold dataset for
```
mlflow.genai.evaluate()
```
or comparable retrieval-quality scorers.

If you only need ad-hoc PDFs (no Databricks workflow), any HTML → PDF tool (
weasyprint
,
wkhtmltopdf
,
playwright pdf
,
plutoprint
) works directly — this skill exists for the synthetic-dataset-on-UC end-to-end shape, not as a general PDF generator.

Path convention:
<SKILL_ROOT>
below = the directory containing this SKILL.md. Resolve to the absolute install path (e.g.
~/.claude/skills/databricks-unstructured-pdf-generation
).
./raw_data/...
paths are relative to your own project cwd.

将HTML文件写入
```
./raw_data/html/
```
（并行写入多个文件以提高速度）——内容领域需与你的检索管道在生产环境中处理的文档匹配。
使用
```
<SKILL_ROOT>/scripts/pdf_generator.py
```
将HTML转换为PDF（并行转换，封装
```
plutoprint
```
）。
通过
```
databricks fs cp
```
将PDF上传至Unity Catalog卷——卷结构需与生产管道读取的结构一致。
生成
```
./raw_data/pdf/pdf_eval_questions.json
```
，将每个文档与检索评估问题配对；该文件将成为
```
mlflow.genai.evaluate()
```
或类似检索质量评分工具的黄金数据集。

如果你只需要临时PDF（无需Databricks工作流），任何HTML转PDF工具（
weasyprint
、
wkhtmltopdf
、
playwright pdf
、
plutoprint
）都可直接使用——本技能的存在是为了实现UC上合成数据集的端到端形态，而非作为通用PDF生成器。

路径约定： 下文的
<SKILL_ROOT>
= 包含本SKILL.md的目录。需解析为绝对安装路径（例如
~/.claude/skills/databricks-unstructured-pdf-generation
）。
./raw_data/...
路径相对于你自己的项目工作目录。

Dependencies

依赖项

bash

uv pip install plutoprint

bash

uv pip install plutoprint

Step 1: Write HTML Files

步骤1：编写HTML文件

bash

mkdir -p ./raw_data/html

Write HTML documents to

./raw_data/html/filename.html

. Use subdirectories to organize (structure is preserved).

bash

mkdir -p ./raw_data/html

将HTML文档写入

./raw_data/html/filename.html

。可使用子目录进行组织（结构会被保留）。

Step 2: Convert to PDF

步骤2：转换为PDF

bash

undefined

bash

undefined

Convert entire folder (parallel, 4 workers)

转换整个文件夹（并行处理，4个工作进程）

python <SKILL_ROOT>/scripts/pdf_generator.py convert --input ./raw_data/html --output ./raw_data/pdf


Skips files where PDF exists and is newer than HTML. Use `--force` to reconvert all.

python <SKILL_ROOT>/scripts/pdf_generator.py convert --input ./raw_data/html --output ./raw_data/pdf


会跳过已存在且比HTML更新的PDF文件。使用`--force`参数可重新转换所有文件。

Step 3: Upload to Volume

步骤3：上传至卷

databricks fs

requires the

dbfs:

scheme prefix even for UC Volume paths.

-r

copies the contents of the source directory into the target (the source directory name is not preserved), so name the target

raw_data/pdf

explicitly to keep the PDFs in their own folder on the volume. They land under

raw_data/pdf/

— i.e.

dbfs:/Volumes/my_catalog/my_schema/raw_data/pdf/report.pdf

— so a Knowledge Assistant or ingest pipeline can point at that single folder.

bash

databricks fs cp -r --overwrite ./raw_data/pdf dbfs:/Volumes/my_catalog/my_schema/raw_data/pdf

databricks fs

要求即使是UC卷路径也需添加

dbfs:

前缀。

-r

参数会将源目录的内容复制到目标目录（不保留源目录名称），因此需显式将目标命名为

raw_data/pdf

，以在卷上保持PDF在独立文件夹中。最终PDF会存放在

raw_data/pdf/

下——即

dbfs:/Volumes/my_catalog/my_schema/raw_data/pdf/report.pdf

——这样知识助手或摄取管道可直接指向该文件夹。

bash

databricks fs cp -r --overwrite ./raw_data/pdf dbfs:/Volumes/my_catalog/my_schema/raw_data/pdf

Step 4: Generate Test Questions

步骤4：生成测试问题

Create

./raw_data/pdf/pdf_eval_questions.json

with questions for Knowledge Assistant (KA) or Multi-Agent Supervisor (MAS) evaluation. It's fine for this file to be uploaded to the volume alongside the PDFs — downstream agents can use it:

json

{
  "api_errors_guide.pdf": {
    "question": "What is the solution for error ERR-4521?",
    "expected_fact": "Call /api/v2/auth/refresh with refresh_token before the 3600s TTL expires"
  },
  "installation_manual.pdf": {
    "question": "What port does the service use by default?",
    "expected_fact": "Port 8443 for HTTPS, configurable via CONFIG_PORT environment variable"
  }
}

This JSON can be used to build KA test cases and validate retrieval accuracy.

创建

./raw_data/pdf/pdf_eval_questions.json

，包含用于知识助手（KA）或多代理管理器（MAS）评估的问题。该文件可与PDF一起上传至卷，供下游代理使用：

json

{
  "api_errors_guide.pdf": {
    "question": "ERR-4521错误的解决方案是什么？",
    "expected_fact": "在3600s的TTL过期前，使用refresh_token调用/api/v2/auth/refresh"
  },
  "installation_manual.pdf": {
    "question": "服务默认使用哪个端口？",
    "expected_fact": "HTTPS默认使用8443端口，可通过CONFIG_PORT环境变量配置"
  }
}

该JSON可用于构建KA测试用例并验证检索准确性。

Document Content Guidelines

文档内容指南

When generating documents for Knowledge Assistant testing or demos:

Multi-page documents: Each PDF should be several pages with substantial content
Specific error codes and solutions: Include product-specific error codes, causes, and resolution steps
Technical details: API endpoints, configuration parameters, version numbers, specific commands
Simple CSS: Keep styling minimal for fast HTML creation and reliable PDF conversion
Queryable facts: Include details a KA must read the document to answer (not general knowledge)

Good document types:

Product user manuals with troubleshooting sections
API error reference guides (error codes, causes, solutions)
Installation/configuration guides with specific steps
Technical specifications with version-specific details

Example content: Instead of generic "Connection failed" errors, write:

"Error ERR-4521: OAuth token expired. Cause: Token TTL exceeded 3600s default. Solution: Call
```
/api/v2/auth/refresh
```
with your refresh_token before expiration. See Section 4.2 for token lifecycle management."

为知识助手测试或演示生成文档时：

多页文档：每个PDF应包含多页且内容充实
特定错误代码及解决方案：包含产品专属错误代码、原因和解决步骤
技术细节：API端点、配置参数、版本号、特定命令
简洁CSS：保持样式简洁，以快速创建HTML并确保PDF转换可靠
可查询事实：包含知识助手必须阅读文档才能回答的细节（而非通用知识）

推荐文档类型：

包含故障排除章节的产品用户手册
API错误参考指南（错误代码、原因、解决方案）
包含具体步骤的安装/配置指南
包含版本专属细节的技术规格说明

示例内容： 不要写通用的“连接失败”错误，应写：

"错误ERR-4521：OAuth令牌过期。原因：令牌TTL超过默认的3600s。解决方案：在过期前使用refresh_token调用
```
/api/v2/auth/refresh
```
。详见4.2节的令牌生命周期管理。"

CLI Reference

CLI参考

python <SKILL_ROOT>/scripts/pdf_generator.py convert [OPTIONS]

  --input, -i     Input HTML file or folder (required)
  --output, -o    Output folder for PDFs (required)
  --force, -f     Force reconvert (ignore timestamps)
  --workers, -w   Parallel workers (default: 4)

python <SKILL_ROOT>/scripts/pdf_generator.py convert [OPTIONS]

  --input, -i     输入HTML文件或文件夹（必填）
  --output, -o    PDF输出文件夹（必填）
  --force, -f     强制重新转换（忽略时间戳）
  --workers, -w   并行工作进程数（默认：4）

Folder Structure

文件夹结构

Subfolder structure is preserved:

./raw_data/html/                    ./raw_data/pdf/
├── report.html             →       ├── report.pdf
├── quarterly/                      ├── quarterly/
│   └── q1.html             →       │   └── q1.pdf
└── legal/                          └── legal/
    └── terms.html          →           └── terms.pdf

子文件夹结构会被保留：

./raw_data/html/                    ./raw_data/pdf/
├── report.html             →       ├── report.pdf
├── quarterly/                      ├── quarterly/
│   └── q1.html             →       │   └── q1.pdf
└── legal/                          └── legal/
    └── terms.html          →           └── terms.pdf

Bundled Script

捆绑脚本

This skill ships one helper script:

File	Description
scripts/pdf_generator.py	HTML → PDF converter (wraps `plutoprint` ); parallel folder conversion with timestamp-skip. Referenced by Step 2 and the CLI Reference.

The script ships at

<SKILL_ROOT>/scripts/pdf_generator.py

. If it is absent, recreate it from the CLI Reference above (a

convert

subcommand taking

--input

--output

--force

--workers

, wrapping

plutoprint

for HTML → PDF).

本技能附带一个辅助脚本：

文件	描述
scripts/pdf_generator.py	HTML转PDF转换器（封装 `plutoprint` ）；支持带时间戳跳过功能的文件夹并行转换。步骤2和CLI参考中均引用了该脚本。

该脚本位于

<SKILL_ROOT>/scripts/pdf_generator.py

。若缺失，可根据上文的CLI参考重新创建（一个包含

--input

--output

--force

--workers

参数的

convert

子命令，封装

plutoprint

实现HTML转PDF）。

Troubleshooting

故障排除

Issue	Solution
"plutoprint not installed"	`uv pip install plutoprint`
PDF looks wrong	Check HTML/CSS syntax
"Volume does not exist"	`databricks volumes create CATALOG SCHEMA VOLUME_NAME MANAGED` (four separate positional args, not `catalog.schema.volume` )

问题	解决方案
"plutoprint not installed"	执行 `uv pip install plutoprint`
PDF显示异常	检查HTML/CSS语法
"Volume does not exist"	执行 `databricks volumes create CATALOG SCHEMA VOLUME_NAME MANAGED` （四个独立的位置参数，而非 `catalog.schema.volume` 格式）