databricks-lakeflow-connect

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Lakeflow Connect

Build managed ingestion pipelines that pull from SaaS apps and databases into Unity Catalog Delta tables, governed end-to-end and powered by serverless Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables / DLT).

Status: mixed catalog — GA connectors for production use, plus Public Preview, Beta, and Private Preview connectors that expand over time. See the connector catalog below.

构建托管式数据摄入管道，从SaaS应用和数据库拉取数据至Unity Catalog Delta表，全程受管控，并由无服务器Lakeflow Spark Declarative Pipelines（前身为Delta Live Tables / DLT）提供支持。

状态： 混合目录——包含可用于生产环境的GA连接器，以及持续扩展的Public Preview、Beta和Private Preview连接器。详见下方连接器目录。

What Is Lakeflow Connect?

什么是Lakeflow Connect？

Managed connectors for ingesting data from SaaS applications and databases. The resulting ingestion pipeline is governed by Unity Catalog and powered by serverless compute and Lakeflow Spark Declarative Pipelines.

Three frames to keep in mind:

Simple and low-maintenance — no client code to write, no message bus to operate; connector + UC Connection + a serverless pipeline.
Unified with the lakehouse — credentials stored in UC, output is governed Delta, runs on Jobs and SDP like any other workload.
Efficient incremental processing — change tracking / CDC / schema evolution / retries are built in.

There are four architecture patterns:

SaaS pull — connector reads from an external SaaS via OAuth or API key, lands in a streaming Delta table.
Database CDC via gateway — an ingestion gateway runs in the customer's network, stages change events to a UC Volume, a serverless ingestion pipeline applies them as CDC into Delta.
Query-based — for sources without native CDC (Oracle / Teradata / SQL Server / PG / MySQL query-based, Snowflake / Redshift / Synapse / BigQuery via Foreign Catalog), the connector issues periodic queries instead of subscribing to a change feed.
Community connectors — template-based, out of scope for this skill.

用于从SaaS应用和数据库摄入数据的托管连接器。生成的摄入管道由Unity Catalog管控，并由无服务器计算和Lakeflow Spark Declarative Pipelines提供支持。

需牢记三个核心特点：

简单易维护 — 无需编写客户端代码，无需运维消息总线；仅需连接器 + UC Connection + 无服务器管道。
与湖仓统一集成 — 凭据存储在UC中，输出为受管控的Delta表，与其他工作负载一样在Jobs和SDP上运行。
高效增量处理 — 内置变更追踪/CDC/ schema演化/重试机制。

共有四种架构模式：

SaaS拉取 — 连接器通过OAuth或API密钥从外部SaaS读取数据，写入流式Delta表。
通过网关实现数据库CDC — 摄入网关在客户网络中运行，将变更事件暂存至UC Volume，无服务器摄入管道将其作为CDC应用至Delta表。
基于查询 — 针对无原生CDC的数据源（Oracle/Teradata/SQL Server/PG/MySQL基于查询；Snowflake/Redshift/Synapse/BigQuery通过Foreign Catalog），连接器定期发起查询而非订阅变更流。
社区连接器 — 基于模板，不属于本技能范畴。

Is Lakeflow Connect the right tool?

Lakeflow Connect是合适的工具吗？

Decide this before you build. Lakeflow Connect is the managed pull path for SaaS apps and databases — it is not the answer for every ingestion intent.

If the source is...	Use	Skill
A SaaS app or database with a managed connector (Salesforce, Workday, ServiceNow, GA4, HubSpot, Confluence, SQL Server, ...)	Lakeflow Connect	this skill
Files on cloud object storage (S3 / ADLS / GCS)	Auto Loader	databricks-pipelines
A source you want to query in place, no copy	Lakehouse Federation	—
An app or device that pushes events at you	Zerobus	databricks-zerobus-ingest
A partner offering a Delta share	Delta Sharing	—

Full reasoning, including the Federation-vs-Connect and Auto-Loader-vs-Connect trade-offs, is in 4-ingestion-decision-tree.md.

在构建前请先确认。Lakeflow Connect是针对SaaS应用和数据库的托管拉取路径——并非适用于所有数据摄入场景。

如果数据源是...	使用工具	对应技能
带有托管连接器的SaaS应用或数据库（Salesforce、Workday、ServiceNow、GA4、HubSpot、Confluence、SQL Server等）	Lakeflow Connect	本技能
云对象存储（S3/ADLS/GCS）上的文件	Auto Loader	databricks-pipelines
需原地查询、无需复制的数据源	Lakehouse Federation	—
主动推送事件的应用或设备	Zerobus	databricks-zerobus-ingest
提供Delta共享的合作伙伴	Delta Sharing	—

完整的决策逻辑，包括Federation与Connect、Auto Loader与Connect的权衡，详见4-ingestion-decision-tree.md。

Connector catalog

连接器目录

Lakeflow Connect ships connectors at multiple release stages. GA and Public Preview connectors are production-supported; Beta and Private Preview are early-access and not production-supported.

Lakeflow Connect提供多个发布阶段的连接器。GA和Public Preview连接器受生产环境支持；Beta和Private Preview为早期访问版本，不受生产环境支持。

GA connectors

GA连接器

Full coverage in this skill.

Source	Type	Auth	Reference
Salesforce (Sales / Service / etc.)	SaaS pull	OAuth U2M	1-saas-connectors.md
Workday Reports (RaaS)	SaaS pull	OAuth refresh token / basic	1-saas-connectors.md
ServiceNow	SaaS pull	OAuth U2M / basic	1-saas-connectors.md
Google Analytics 4	SaaS pull (via BigQuery)	Service-account JSON	1-saas-connectors.md
HubSpot	SaaS pull	OAuth	1-saas-connectors.md
Confluence	SaaS pull	OAuth	1-saas-connectors.md
SQL Server (cloud)	Database CDC	DB user + change tracking / CDC	2-database-connectors.md
SQL Server (on-prem)	Database CDC	DB user + ExpressRoute / Direct Connect	2-database-connectors.md

本技能提供完整说明。

数据源	类型	认证方式	参考文档
Salesforce（销售/服务等）	SaaS拉取	OAuth U2M	1-saas-connectors.md
Workday Reports（RaaS）	SaaS拉取	OAuth刷新令牌/基础认证	1-saas-connectors.md
ServiceNow	SaaS拉取	OAuth U2M/基础认证	1-saas-connectors.md
Google Analytics 4	SaaS拉取（通过BigQuery）	服务账号JSON	1-saas-connectors.md
HubSpot	SaaS拉取	OAuth	1-saas-connectors.md
Confluence	SaaS拉取	OAuth	1-saas-connectors.md
SQL Server（云）	数据库CDC	DB用户+变更追踪/CDC	2-database-connectors.md
SQL Server（本地）	数据库CDC	DB用户+ExpressRoute/Direct Connect	2-database-connectors.md

Public Preview connectors

Public Preview连接器

Production-supported. Configuration may evolve before GA. Deep coverage is being added incrementally; until then, see the public connector reference for current setup steps.

Source	Type	Auth
NetSuite	SaaS pull	OAuth
Dynamics 365	SaaS pull	OAuth
PostgreSQL CDC	Database CDC	DB user + gateway
MySQL CDC	Database CDC	DB user + gateway
Oracle / Teradata / SQL Server / PG / MySQL (query-based)	Database query	DB user
Snowflake / Redshift / Synapse / BigQuery (Foreign Catalog)	Database query	Foreign Catalog
SFTP	File pull	Key / password

受生产环境支持。配置可能在GA前有所变动。详细说明正在逐步完善；在此之前，请查看公共连接器参考文档获取当前设置步骤。

数据源	类型	认证方式
NetSuite	SaaS拉取	OAuth
Dynamics 365	SaaS拉取	OAuth
PostgreSQL CDC	数据库CDC	DB用户+网关
MySQL CDC	数据库CDC	DB用户+网关
Oracle/Teradata/SQL Server/PG/MySQL（基于查询）	数据库查询	DB用户
Snowflake/Redshift/Synapse/BigQuery（Foreign Catalog）	数据库查询	Foreign Catalog
SFTP	文件拉取	密钥/密码

Beta and Private Preview

Beta和Private Preview

Early-access connectors are not production-supported. The list changes month to month; check the public connector reference for current availability.

For the Lakeflow-Connect-vs-Auto-Loader-vs-Federation-vs-Delta-Sharing decision, see 4-ingestion-decision-tree.md.

早期访问连接器不受生产环境支持。列表每月更新；请查看公共连接器参考文档获取当前可用连接器。

关于Lakeflow Connect与Auto Loader、Federation、Delta Sharing的决策，详见4-ingestion-decision-tree.md。

Required Tools

必备工具

Databricks CLI v0.294.0+ for

databricks pipelines create

and

databricks connections create

. Verify with

databricks --version

Databricks SDK for Python (
```
databricks-sdk>=0.85.0
```
) if you prefer SDK over CLI.
Declarative Automation Bundles if authoring as IaC (recommended for any pipeline that ships to a customer environment).

No extra connector-specific SDK is needed. Lakeflow Connect reuses the pipelines API surface — pipelines are created with an

ingestion_definition

block instead of a

libraries

block, but the API and CLI are otherwise the same.

Databricks CLI v0.294.0+：用于执行

databricks pipelines create

和

databricks connections create

命令。可通过

databricks --version

验证版本。

Databricks SDK for Python（
```
databricks-sdk>=0.85.0
```
）：若偏好使用SDK而非CLI。
Declarative Automation Bundles：若以IaC方式编写（推荐用于任何部署至客户环境的管道）。

无需额外的连接器专用SDK。Lakeflow Connect复用pipelines API接口——创建管道时使用

ingestion_definition

块而非

libraries

块，但API和CLI的其他部分保持一致。

Prerequisites

前置条件

Confirm before creating any pipeline:

A Unity Catalog target — catalog and schema must exist; the service principal or user creating the pipeline needs
```
USE CATALOG
```
,
```
USE SCHEMA
```
,
```
CREATE TABLE
```
, and
```
MODIFY
```
on the target schema.
A UC
CONNECTION
object with credentials for the source. SaaS OAuth U2M connections must be created via the UI (Catalog Explorer); API-key and basic-auth connections can be created via CLI / DAB.
For database connectors: network reachability between the gateway (classic compute, customer VPC) and the source database. On-prem requires ExpressRoute (Azure) or Direct Connect (AWS).
For file connectors: OAuth scope grants on the SaaS file repo (SharePoint / Google Drive).

创建任何管道前请确认：

Unity Catalog目标 — 目录和Schema必须已存在；创建管道的服务主体或用户需拥有目标Schema的
```
USE CATALOG
```
、
```
USE SCHEMA
```
、
```
CREATE TABLE
```
和
```
MODIFY
```
权限。
UC
CONNECTION
对象：包含数据源的凭据。SaaS OAuth U2M连接必须通过UI（Catalog Explorer）创建；API密钥和基础认证连接可通过CLI/DAB创建。
针对数据库连接器：网关（经典计算，客户VPC）与源数据库之间的网络可达性。本地数据库需要ExpressRoute（Azure）或Direct Connect（AWS）。
针对文件连接器：SaaS文件仓库（SharePoint/Google Drive）的OAuth权限授予。

Minimal Example — Salesforce ingestion pipeline

最简示例——Salesforce摄入管道

The canonical authoring path is JSON to

databricks pipelines create --json

. (There is no SQL

CREATE TABLE … FROM CONNECTION

syntax for Lakeflow Connect — that syntax exists only for Lakehouse Federation, which is a different product.)

bash

databricks pipelines create --json '{
  "name": "salesforce_to_uc",
  "ingestion_definition": {
    "connection_name": "my_salesforce_oauth_connection",
    "objects": [
      {"table": {"source_schema": "salesforce", "source_table": "Account",
                 "destination_catalog": "main", "destination_schema": "salesforce_raw"}},
      {"table": {"source_schema": "salesforce", "source_table": "Opportunity",
                 "destination_catalog": "main", "destination_schema": "salesforce_raw"}}
    ]
  }
}'

For a DAB-authored version (the production path), see 1-saas-connectors.md.

标准编写方式是将JSON传入

databricks pipelines create --json

命令。（Lakeflow Connect没有SQL

CREATE TABLE … FROM CONNECTION

语法——该语法仅适用于Lakehouse Federation，是另一款产品。）

bash

databricks pipelines create --json '{
  "name": "salesforce_to_uc",
  "ingestion_definition": {
    "connection_name": "my_salesforce_oauth_connection",
    "objects": [
      {"table": {"source_schema": "salesforce", "source_table": "Account",
                 "destination_catalog": "main", "destination_schema": "salesforce_raw"}},
      {"table": {"source_schema": "salesforce", "source_table": "Opportunity",
                 "destination_catalog": "main", "destination_schema": "salesforce_raw"}}
    ]
  }
}'

关于DAB编写版本（生产环境推荐方式），详见1-saas-connectors.md。

Running the pipeline

运行管道

Once authored, deploy and trigger a run. The bundle path gives the cleanest run-by-key command:

bash

databricks bundle deploy -t dev
databricks bundle run salesforce_ingestion           # KEY = the pipeline resource key in the bundle; waits by default
databricks bundle run salesforce_ingestion --no-wait

A pipeline created imperatively with

pipelines create --json

has no run-by-name CLI — start and poll an update by pipeline ID instead:

bash

databricks pipelines start-update <pipeline-id>             # returns an update_id
databricks pipelines get-update  <pipeline-id> <update-id>  # poll one update's status
databricks pipelines list-updates <pipeline-id>             # recent updates and their states

That asymmetry is one more reason to author with a Declarative Automation Bundle.

编写完成后，部署并触发运行。Bundle路径提供最简洁的按密钥运行命令：

bash

databricks bundle deploy -t dev
databricks bundle run salesforce_ingestion           # KEY = bundle中的管道资源密钥；默认等待运行完成
databricks bundle run salesforce_ingestion --no-wait

通过

pipelines create --json

临时创建的管道没有按名称运行的CLI命令——需通过管道ID启动并轮询更新：

bash

databricks pipelines start-update <pipeline-id>             # 返回update_id
databricks pipelines get-update  <pipeline-id> <update-id>  # 轮询单个更新的状态
databricks pipelines list-updates <pipeline-id>             # 查看最近的更新及其状态

这种不对称性是使用Declarative Automation Bundle编写管道的又一原因。

Detailed guides

详细指南

Topic	File	When to read
SaaS connectors (Salesforce, Workday Reports, ServiceNow, GA4, HubSpot, Confluence)	1-saas-connectors.md	Unified SaaS pattern, per-connector deltas, OAuth flows, DAB stubs
Database connectors (SQL Server cloud + on-prem)	2-database-connectors.md	Gateway pattern, change tracking vs CDC, network setup
Ingestion decision tree	4-ingestion-decision-tree.md	Lakeflow Connect vs Auto Loader vs Lakehouse Federation vs Delta Sharing
Troubleshooting and monitoring	5-troubleshooting-and-monitoring.md	Event log queries, common errors, escalation pointers

主题	文件	阅读时机
SaaS连接器（Salesforce、Workday Reports、ServiceNow、GA4、HubSpot、Confluence）	1-saas-connectors.md	统一SaaS模式、各连接器差异、OAuth流程、DAB模板
数据库连接器（云及本地SQL Server）	2-database-connectors.md	网关模式、变更追踪vs CDC、网络设置
摄入决策树	4-ingestion-decision-tree.md	Lakeflow Connect vs Auto Loader vs Lakehouse Federation vs Delta Sharing
故障排查与监控	5-troubleshooting-and-monitoring.md	事件日志查询、常见错误、升级指引

Workflow

工作流程

For each new ingestion pipeline:

Pick the connector category — SaaS / database / file / push — and read the matching reference file.
Verify prerequisites — UC target, source credentials, network path (for databases), region availability.
Create the UC
CONNECTION
— UI for OAuth U2M, CLI / DAB for everything else.
Author the pipeline —
```
databricks pipelines create --json
```
for one-offs, DAB YAML for anything shipping to a customer.
Trigger the first run and watch the event log; see 5-troubleshooting-and-monitoring.md for the SQL.
Schedule the triggered pipeline with a Jobs
```
pipeline_task
```
(cron or interval). Lakeflow Connect supports triggered runs only —
```
continuous: false
```
selects triggered mode but is not itself a schedule, so the cadence comes from the Jobs trigger.

针对每个新的摄入管道：

选择连接器类别 — SaaS/数据库/文件/推送，并阅读对应的参考文档。
验证前置条件 — UC目标、源凭据、网络路径（针对数据库）、区域可用性。
创建UC
CONNECTION
— OAuth U2M通过UI创建，其他方式通过CLI/DAB创建。
编写管道 — 一次性任务使用
```
databricks pipelines create --json
```
，部署至客户环境的任务使用DAB YAML。
触发首次运行并查看事件日志；故障排查SQL详见5-troubleshooting-and-monitoring.md。
调度：通过Jobs的
```
pipeline_task
```
（ cron或间隔）触发管道。Lakeflow Connect仅支持触发式运行——
```
continuous: false
```
选择触发模式但本身不是调度，因此调度周期由Jobs触发器决定。

Anti-patterns

反模式

Three forms that look plausible but fail — wrong vs. right:

1.
CREATE TABLE ... FROM CONNECTION
is Lakehouse Federation, not Lakeflow Connect.

sql

-- WRONG: Federation syntax; no LFC equivalent exists
CREATE TABLE main.salesforce_raw.account FROM CONNECTION my_salesforce_conn;

json

// RIGHT: author an ingestion_definition (see the Minimal Example above)
{"ingestion_definition": {"connection_name": "my_salesforce_conn", "objects": [/* ... */]}}

2. An ingestion pipeline carries
ingestion_definition
, never a
libraries
block.

json

// WRONG: libraries is for a standard SDP pipeline running your notebooks/files
{"name": "salesforce_to_uc", "libraries": [{"notebook": {"path": "/Repos/.../ingest"}}]}
// RIGHT:
{"name": "salesforce_to_uc", "ingestion_definition": {"connection_name": "...", "objects": []}}

3.
continuous: true
is rejected — Lakeflow Connect is triggered-only.

json

// WRONG: continuous mode fails at create
{"continuous": true, "ingestion_definition": {/* ... */}}
// RIGHT: continuous:false (or omit) + schedule with a Jobs pipeline_task
{"continuous": false, "ingestion_definition": {/* ... */}}

三种看似合理但会失败的方式——错误vs正确：

1.
CREATE TABLE ... FROM CONNECTION
是Lakehouse Federation语法，而非Lakeflow Connect。

sql

-- 错误：Federation语法；Lakeflow Connect无等效语法
CREATE TABLE main.salesforce_raw.account FROM CONNECTION my_salesforce_conn;

json

// 正确：编写ingestion_definition（见上方最简示例）
{"ingestion_definition": {"connection_name": "my_salesforce_conn", "objects": [/* ... */]}}

2. 摄入管道包含
ingestion_definition
，绝不能包含
libraries
块。

json

// 错误：libraries用于运行您的笔记本/文件的标准SDP管道
{"name": "salesforce_to_uc", "libraries": [{"notebook": {"path": "/Repos/.../ingest"}}]}
// 正确:
{"name": "salesforce_to_uc", "ingestion_definition": {"connection_name": "...", "objects": []}}

3.
continuous: true
会被拒绝——Lakeflow Connect仅支持触发式运行。

json

// 错误：创建时会失败
{"continuous": true, "ingestion_definition": {/* ... */}}
// 正确：continuous:false（或省略） + 通过Jobs pipeline_task调度
{"continuous": false, "ingestion_definition": {/* ... */}}

Important

重要说明

Triggered only, no continuous mode — pipelines run on a schedule or on-demand, never continuously. Check the connector reference for the latest status.
Compute-only billing — Lakeflow Connect is billed in DBUs (no per-row fee). Database connectors also incur classic-compute gateway DBUs in addition to the serverless ingestion pipeline DBUs. See the pricing page for current rates.
Salesforce auth is OAuth U2M only — no machine-to-machine, no basic auth. Connection creation requires a UI walk-through.
Database staging retention is 30 days by default in the UC Volume between the gateway and the ingestion pipeline.
Limits per pipeline — most SaaS connectors cap at 250 tables per pipeline. Split across multiple pipelines if needed.
This lands raw tables — Lakeflow Connect writes source-faithful tables (the ingestion landing zone). Build the medallion Bronze/Silver/Gold transforms on top of them with databricks-pipelines.

仅支持触发式运行，无持续模式 — 管道按调度或按需运行，从不持续运行。请查看连接器参考文档获取最新状态。
仅按计算量计费 — Lakeflow Connect按DBU计费（无按行收费）。数据库连接器除无服务器摄入管道的DBU外，还会产生经典计算网关的DBU。当前费率详见定价页面。
Salesforce仅支持OAuth U2M认证 — 无机器对机器认证，无基础认证。连接创建需要通过UI完成。
数据库暂存数据默认保留30天 — 在网关与摄入管道之间的UC Volume中。
单管道限制 — 大多数SaaS连接器限制单管道最多250张表。如有需要，请拆分至多个管道。
生成原始表 — Lakeflow Connect写入与源一致的表（摄入着陆区）。需在其之上使用databricks-pipelines构建Bronze/Silver/Gold的medallion转换。

Key Concepts

核心概念

UC
CONNECTION
is the credential anchor — every Lakeflow Connect pipeline points at a UC connection. The connection owns the auth; the pipeline references it by name.
Serverless ingestion pipeline + (optional) classic gateway — SaaS connectors are pure serverless. Database connectors split into a customer-network gateway (classic) and a serverless ingestion pipeline (Delta-bound).
CDC and schema evolution are built in — for sources that support change tracking or CDC, the connector applies changes incrementally and evolves the target schema. Data-type changes typically require a full snapshot reload.
Streaming Delta output — destination tables are governed Delta tables; CDC sources are applied with change semantics (
```
APPLY CHANGES
```
/ AUTO CDC, or
```
apply_changes_from_snapshot
```
for snapshot sources). Compatible with downstream materialized views and Spark streaming.
OAuth U2M is UI-only — DAB / CLI cannot bootstrap OAuth U2M connections. Hand the one-time browser step to a human (Catalog Explorer > External Data > Connections > Create connection > pick the source > sign in), then resume once
```
databricks connections get <connection_name>
```
reports
```
READY
```
.

UC
CONNECTION
是凭据锚点 — 每个Lakeflow Connect管道都指向一个UC连接。连接管理认证，管道通过名称引用它。
无服务器摄入管道 +（可选）经典网关 — SaaS连接器纯无服务器。数据库连接器分为客户网络网关（经典）和无服务器摄入管道（写入Delta）两部分。
内置CDC和schema演化 — 针对支持变更追踪或CDC的数据源，连接器增量应用变更并演化目标schema。数据类型变更通常需要重新加载完整快照。
流式Delta输出 — 目标表为受管控的Delta表；CDC数据源通过变更语义（
```
APPLY CHANGES
```
/AUTO CDC，或针对快照源的
```
apply_changes_from_snapshot
```
）应用。与下游物化视图和Spark流式处理兼容。
OAuth U2M仅支持UI创建 — DAB/CLI无法初始化OAuth U2M连接。需由人工完成一次性浏览器步骤（Catalog Explorer > External Data > Connections > Create connection > 选择数据源 > 登录），待
```
databricks connections get <connection_name>
```
返回
```
READY
```
后继续。

Common Issues

常见问题

For common errors and their fixes — duplicate-key violations, watermark / cursor problems, schema evolution, gateway region availability, the

channel

runtime-channel setting, and pipelines that run but land no data — see 5-troubleshooting-and-monitoring.md, which also has the event-log queries to diagnose them.

关于常见错误及其修复——重复键冲突、水印/游标问题、schema演化、网关区域可用性、

channel

运行时通道设置、运行但无数据写入的管道等，详见5-troubleshooting-and-monitoring.md，其中还包含用于诊断问题的事件日志查询语句。

databricks-lakeflow-connect

Original

Translation

Lakeflow Connect

Lakeflow Connect

What Is Lakeflow Connect?

什么是Lakeflow Connect？

Is Lakeflow Connect the right tool?

Lakeflow Connect是合适的工具吗？

Connector catalog

连接器目录

GA connectors

GA连接器

Public Preview connectors

Public Preview连接器

Beta and Private Preview

Beta和Private Preview

Required Tools

必备工具

Prerequisites

前置条件

Minimal Example — Salesforce ingestion pipeline

最简示例——Salesforce摄入管道

Running the pipeline

运行管道

Detailed guides

详细指南

Workflow

工作流程

Anti-patterns

反模式

Important

重要说明

Key Concepts

核心概念

Common Issues

常见问题

Related Skills

相关技能

Resources

资源