databricks-iceberg

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Apache Iceberg on Databricks

Databricks 上的 Apache Iceberg

Databricks provides multiple ways to work with Apache Iceberg: native managed Iceberg tables, UniForm for Delta-to-Iceberg interoperability, and the Iceberg REST Catalog (IRC) for external engine access.

Databricks 提供多种与 Apache Iceberg 协作的方式：原生托管 Iceberg 表、用于 Delta 与 Iceberg 互操作的 UniForm，以及用于外部引擎访问的 Iceberg REST Catalog（IRC）。

Critical Rules (always follow)

关键规则（必须遵守）

MUST use Unity Catalog — all Iceberg features require UC-enabled workspaces
MUST NOT install an Iceberg library into Databricks Runtime (DBR includes built-in Iceberg support; adding a library causes version conflicts)
MUST NOT set
```
write.metadata.path
```
or
```
write.metadata.previous-versions-max
```
— Databricks manages metadata locations automatically; overriding causes corruption
MUST determine which Iceberg pattern fits the use case before writing code — see the When to Use section below
MUST know that both
```
PARTITIONED BY
```
and
```
CLUSTER BY
```
produce the same Iceberg metadata for external engines — UC maintains an Iceberg partition spec with partition fields corresponding to the clustering keys, so external engines reading via IRC see a partitioned Iceberg table (not Hive-style, but proper Iceberg partition fields) and can prune on those fields; internally UC uses those fields as liquid clustering keys; the only differences between the two syntaxes are: (1)
```
PARTITIONED BY
```
is standard Iceberg DDL (any engine can create the table), while
```
CLUSTER BY
```
is DBR-only DDL; (2)
```
PARTITIONED BY
```
auto-handles DV/row-tracking properties, while
```
CLUSTER BY
```
requires manual TBLPROPERTIES on v2
MUST NOT use expression-based partition transforms (
```
bucket()
```
,
```
years()
```
,
```
months()
```
,
```
days()
```
,
```
hours()
```
) with
```
PARTITIONED BY
```
on managed Iceberg tables — only plain column references are supported; expression transforms cause errors
MUST disable deletion vectors and row tracking when using
```
CLUSTER BY
```
on Iceberg v2 tables — set
```
'delta.enableDeletionVectors' = false
```
and
```
'delta.enableRowTracking' = false
```
in TBLPROPERTIES (Iceberg v3 handles this automatically;
```
PARTITIONED BY
```
handles this automatically on both v2 and v3)

必须使用 Unity Catalog —— 所有 Iceberg 功能均要求工作区已启用 UC
禁止在 Databricks Runtime（DBR）中安装 Iceberg 库 —— DBR 已内置 Iceberg 支持；添加库会导致版本冲突
禁止设置
```
write.metadata.path
```
或
```
write.metadata.previous-versions-max
```
—— Databricks 会自动管理元数据位置；手动覆盖会导致数据损坏
必须在编写代码前确定适合用例的 Iceberg 模式 —— 请参阅下方的【适用场景】部分
必须了解
```
PARTITIONED BY
```
和
```
CLUSTER BY
```
为外部引擎生成的 Iceberg 元数据是相同的 —— UC 会维护一个 Iceberg 分区规范，其中分区字段与聚类键对应，因此通过 IRC 读取的外部引擎会看到一个分区化的 Iceberg 表（不是 Hive 风格，而是标准的 Iceberg 分区字段），并可对这些字段进行分区裁剪；UC 内部将这些字段用作液态聚类键；两种语法的唯一区别是：(1)
```
PARTITIONED BY
```
是标准 Iceberg DDL（任何引擎均可创建表），而
```
CLUSTER BY
```
是 DBR 专属 DDL；(2)
```
PARTITIONED BY
```
会自动处理 DV/行跟踪属性，而
```
CLUSTER BY
```
在 v2 版本中需要手动设置 TBLPROPERTIES
禁止在托管 Iceberg 表的
```
PARTITIONED BY
```
中使用基于表达式的分区转换（
```
bucket()
```
、
```
years()
```
、
```
months()
```
、
```
days()
```
、
```
hours()
```
）—— 仅支持纯列引用；表达式转换会导致错误
必须在 Iceberg v2 表上使用
```
CLUSTER BY
```
时禁用删除向量和行跟踪 —— 在 TBLPROPERTIES 中设置
```
'delta.enableDeletionVectors' = false
```
和
```
'delta.enableRowTracking' = false
```
（Iceberg v3 会自动处理此问题；
```
PARTITIONED BY
```
在 v2 和 v3 版本中均会自动处理此问题）

Key Concepts

核心概念

Concept	Summary
Managed Iceberg Table	Native Iceberg table created with `USING ICEBERG` — full read/write in Databricks and via external Iceberg engines
External Iceberg Reads (Uniform)	Delta table that auto-generates Iceberg metadata — read as Iceberg externally, write as Delta internally
Compatibility Mode	UniForm variant for streaming tables and materialized views in SDP pipelines
Iceberg REST Catalog (IRC)	Unity Catalog's built-in REST endpoint implementing the Iceberg REST Catalog spec — lets external engines (Spark, PyIceberg, Snowflake) access UC-managed Iceberg data
Iceberg v3	Next-gen format (Beta, DBR 17.3+) — deletion vectors, VARIANT type, row lineage

概念	概述
托管 Iceberg 表	使用 `USING ICEBERG` 创建的原生 Iceberg 表——可在 Databricks 及外部 Iceberg 引擎中进行完整的读写操作
外部 Iceberg 读取（Uniform）	自动生成 Iceberg 元数据的 Delta 表——外部可作为 Iceberg 读取，内部作为 Delta 写入
兼容模式	适用于 SDP 流水线中流表和物化视图的 UniForm 变体
Iceberg REST Catalog（IRC）	Unity Catalog 内置的 REST 端点，实现了 Iceberg REST Catalog 规范——允许外部引擎（Spark、PyIceberg、Snowflake）访问 UC 托管的 Iceberg 数据
Iceberg v3	下一代格式（Beta 版，DBR 17.3+）——支持删除向量、VARIANT 类型、行谱系

Quick Start

快速开始

Create a Managed Iceberg Table

创建托管 Iceberg 表

sql

-- No clustering
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
AS SELECT * FROM raw_events;

-- PARTITIONED BY (recommended for cross-platform): standard Iceberg syntax, works on EMR/OSS Spark/Trino/Flink
-- auto-disables DVs and row tracking — no TBLPROPERTIES needed on v2 or v3
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
PARTITIONED BY (event_date)
AS SELECT * FROM raw_events;

-- CLUSTER BY on Iceberg v2 (DBR-only syntax): must manually disable DVs and row tracking
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
TBLPROPERTIES (
  'delta.enableDeletionVectors' = false,
  'delta.enableRowTracking' = false
)
CLUSTER BY (event_date)
AS SELECT * FROM raw_events;

-- CLUSTER BY on Iceberg v3 (DBR-only syntax): no TBLPROPERTIES needed
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
TBLPROPERTIES ('format-version' = '3')
CLUSTER BY (event_date)
AS SELECT * FROM raw_events;

sql

-- No clustering
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
AS SELECT * FROM raw_events;

-- PARTITIONED BY (recommended for cross-platform): standard Iceberg syntax, works on EMR/OSS Spark/Trino/Flink
-- auto-disables DVs and row tracking — no TBLPROPERTIES needed on v2 or v3
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
PARTITIONED BY (event_date)
AS SELECT * FROM raw_events;

-- CLUSTER BY on Iceberg v2 (DBR-only syntax): must manually disable DVs and row tracking
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
TBLPROPERTIES (
  'delta.enableDeletionVectors' = false,
  'delta.enableRowTracking' = false
)
CLUSTER BY (event_date)
AS SELECT * FROM raw_events;

-- CLUSTER BY on Iceberg v3 (DBR-only syntax): no TBLPROPERTIES needed
CREATE TABLE my_catalog.my_schema.events
USING ICEBERG
TBLPROPERTIES ('format-version' = '3')
CLUSTER BY (event_date)
AS SELECT * FROM raw_events;

Enable UniForm on an Existing Delta Table

在现有 Delta 表上启用 UniForm

sql

ALTER TABLE my_catalog.my_schema.customers
SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

sql

ALTER TABLE my_catalog.my_schema.customers
SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

Read/Write Capability Matrix

读写能力矩阵

Table Type	Databricks Read	Databricks Write	External IRC Read	External IRC Write
Managed Iceberg ( `USING ICEBERG` )	Yes	Yes	Yes	Yes
Delta + UniForm	Yes (as Delta)	Yes (as Delta)	Yes (as Iceberg)	No
Delta + Compatibility Mode	Yes (as Delta)	Yes	Yes (as Iceberg)	No

表类型	Databricks 读取	Databricks 写入	外部 IRC 读取	外部 IRC 写入
托管 Iceberg（ `USING ICEBERG` ）	是	是	是	是
Delta + UniForm	是（作为 Delta）	是（作为 Delta）	是（作为 Iceberg）	否
Delta + 兼容模式	是（作为 Delta）	是	是（作为 Iceberg）	否

Reference Files

参考文档

File	Summary	Keywords
references/1-managed-iceberg-tables.md	Creating and managing native Iceberg tables — DDL, DML, Liquid Clustering, Predictive Optimization, Iceberg v3, limitations	CREATE TABLE USING ICEBERG, CTAS, MERGE, time travel, deletion vectors, VARIANT
references/2-uniform-and-compatibility.md	Making Delta tables readable as Iceberg — UniForm for regular tables, Compatibility Mode for streaming tables and MVs	UniForm, universalFormat, Compatibility Mode, streaming tables, materialized views, SDP
references/3-iceberg-rest-catalog.md	Exposing Databricks tables to external engines via the IRC endpoint — auth, credential vending, IP access lists	IRC, REST Catalog, credential vending, EXTERNAL USE SCHEMA, PAT, OAuth
references/4-snowflake-interop.md	Bidirectional Snowflake-Databricks integration — catalog integration, foreign catalogs, vended credentials	Snowflake, catalog integration, external volume, vended credentials, REFRESH_INTERVAL_SECONDS
references/5-external-engine-interop.md	Connecting PyIceberg, OSS Spark, AWS EMR, Apache Flink, and Kafka Connect via IRC	PyIceberg, OSS Spark, EMR, Flink, Kafka Connect, pyiceberg.yaml

文档	概述	关键词
references/1-managed-iceberg-tables.md	创建和管理原生 Iceberg 表——DDL、DML、液态聚类、预测优化、Iceberg v3、限制	CREATE TABLE USING ICEBERG、CTAS、MERGE、时间旅行、删除向量、VARIANT
references/2-uniform-and-compatibility.md	将 Delta 表转换为可作为 Iceberg 读取的表——针对普通表的 UniForm、针对流表和物化视图的兼容模式	UniForm、universalFormat、兼容模式、流表、物化视图、SDP
references/3-iceberg-rest-catalog.md	通过 IRC 端点向外部引擎暴露 Databricks 表——认证、凭证分发、IP 访问列表	IRC、REST Catalog、凭证分发、EXTERNAL USE SCHEMA、PAT、OAuth
references/4-snowflake-interop.md	Snowflake 与 Databricks 的双向集成——目录集成、外部目录、凭证分发	Snowflake、目录集成、外部卷、凭证分发、REFRESH_INTERVAL_SECONDS
references/5-external-engine-interop.md	通过 IRC 连接 PyIceberg、OSS Spark、AWS EMR、Apache Flink 和 Kafka Connect	PyIceberg、OSS Spark、EMR、Flink、Kafka Connect、pyiceberg.yaml

When to Use

适用场景

Creating a new Iceberg table → references/1-managed-iceberg-tables.md
Making an existing Delta table readable as Iceberg → references/2-uniform-and-compatibility.md
Making a streaming table or MV readable as Iceberg → references/2-uniform-and-compatibility.md (Compatibility Mode section)
Choosing between Managed Iceberg vs UniForm vs Compatibility Mode → decision table in references/2-uniform-and-compatibility.md
Exposing Databricks tables to external engines via REST API → references/3-iceberg-rest-catalog.md
Integrating Databricks with Snowflake (either direction) → references/4-snowflake-interop.md
Connecting PyIceberg, OSS Spark, Flink, EMR, or Kafka → references/5-external-engine-interop.md

创建新的 Iceberg 表 → references/1-managed-iceberg-tables.md
将现有 Delta 表转换为可作为 Iceberg 读取的表 → references/2-uniform-and-compatibility.md
将流表或物化视图转换为可作为 Iceberg 读取的表 → references/2-uniform-and-compatibility.md（兼容模式部分）
选择托管 Iceberg、UniForm 还是兼容模式 → references/2-uniform-and-compatibility.md 中的决策表
通过 REST API 向外部引擎暴露 Databricks 表 → references/3-iceberg-rest-catalog.md
Databricks 与 Snowflake 集成（双向） → references/4-snowflake-interop.md
连接 PyIceberg、OSS Spark、Flink、EMR 或 Kafka → references/5-external-engine-interop.md

Common Issues

常见问题

Issue	Solution
No Change Data Feed (CDF)	CDF is not supported on managed Iceberg tables. Use Delta + UniForm if you need CDF.
UniForm async delay	Iceberg metadata generation is asynchronous. After a write, there may be a brief delay before external engines see the latest data. Check status with `DESCRIBE EXTENDED table_name` .
Compression codec change	Managed Iceberg tables use `zstd` compression by default (not `snappy` ). Older Iceberg readers that don't support zstd will fail. Verify reader compatibility or set `write.parquet.compression-codec` to `snappy` .
Snowflake 1000-commit limit	Snowflake's Iceberg catalog integration can only see the last 1000 Iceberg commits. High-frequency writers must compact metadata or Snowflake will lose visibility of older data.
Deletion vectors with UniForm	UniForm requires deletion vectors to be disabled ( `delta.enableDeletionVectors = false` ). If your table has deletion vectors enabled, disable them before enabling UniForm.
No shallow clone for Iceberg	`SHALLOW CLONE` is not supported for Iceberg tables. Use `DEEP CLONE` or `CREATE TABLE ... AS SELECT` instead.
Version mismatch with external engines	Ensure external engines use an Iceberg library version compatible with the format version of your tables. Iceberg v3 tables require Iceberg library 1.9.0+.

问题	解决方案
无变更数据捕获（CDF）	托管 Iceberg 表不支持 CDF。如果需要 CDF，请使用 Delta + UniForm。
UniForm 异步延迟	Iceberg 元数据生成是异步的。写入后，外部引擎可能需要短暂延迟才能看到最新数据。使用 `DESCRIBE EXTENDED table_name` 检查状态。
压缩编解码器变更	托管 Iceberg 表默认使用 `zstd` 压缩（而非 `snappy` ）。不支持 zstd 的旧版 Iceberg 读取器会失败。请验证读取器兼容性，或设置 `write.parquet.compression-codec` 为 `snappy` 。
Snowflake 1000 次提交限制	Snowflake 的 Iceberg 目录集成只能查看最近 1000 次 Iceberg 提交。高频写入者必须压缩元数据，否则 Snowflake 会丢失旧数据的可见性。
UniForm 与删除向量冲突	UniForm 要求禁用删除向量（ `delta.enableDeletionVectors = false` ）。如果表已启用删除向量，请先禁用再启用 UniForm。
Iceberg 不支持浅克隆	Iceberg 表不支持 `SHALLOW CLONE` 。请改用 `DEEP CLONE` 或 `CREATE TABLE ... AS SELECT` 。
与外部引擎版本不匹配	确保外部引擎使用的 Iceberg 库版本与表的格式版本兼容。Iceberg v3 表需要 Iceberg 库 1.9.0+。

Related Skills

Resources

资源

Iceberg Overview — main hub for Iceberg on Databricks
UniForm — Delta Universal Format
Iceberg REST Catalog — IRC endpoint and external engine access
Compatibility Mode — UniForm for streaming tables and MVs
Iceberg v3 — next-gen format features (Beta)
Foreign Tables — reading external catalog data

Iceberg 概述 —— Databricks 上 Iceberg 的主 hub
UniForm —— Delta 通用格式
Iceberg REST Catalog —— IRC 端点与外部引擎访问
兼容模式 —— 适用于流表和物化视图的 UniForm
Iceberg v3 —— 下一代格式功能（Beta 版）
外部表 —— 读取外部目录数据