Loading...
Loading...
Lakeflow Connect ingestion from SaaS apps (Salesforce, Workday, ServiceNow, HubSpot, SharePoint, Google Drive, OneDrive, Jira, ads, audit logs, 70+ more) and Kafka or RabbitMQ into Unity Catalog. Use when finding a connector, creating its connection and pipeline, scheduling or debugging it.
npx skill4agent add databricks/databricks-agent-skills databricks-lakeflow-connect-saasdatabricks-core| Source | Skill |
|---|---|
| A SaaS application, a RabbitMQ queue, or a Kafka topic with the managed connector (Beta) | this skill |
| A Kafka topic whose messages need more than the key and value formats the Kafka pipeline page lists | |
| A database (SQL Server, PostgreSQL, MySQL, Oracle, query-based) | databricks-lakeflow-connect |
| An app or device that pushes events | databricks-zerobus-ingest |
| Files on cloud object storage | Auto Loader, databricks-pipelines |
| Files on an SFTP server | SFTP file connector on Auto Loader: file-connectors.md |
| A stream with no managed connector (Kinesis, Pub/Sub, Pulsar) | databricks-spark-structured-streaming |
| A source with no managed connector | a custom connector (Beta) or a community connector |
https://docs.databricks.com/ingestion/lakeflow-connect/<key><key>-overview-source-setup-connection-pipeline-reference-limits-troubleshoot-reference-limits-source-setup-connection-connectionconnection_typedatabricks connections create-pipelinesource_schemadefaultobjectspipeline_taskpipelinesingestion_definitionpipeline_task{
"name": "jira_ingest",
"catalog": "main",
"schema": "jira_raw",
"ingestion_definition": {
"connection_name": "<jira-connection>",
"objects": [
{"table": {"source_schema": "default", "source_table": "issues",
"destination_catalog": "main", "destination_schema": "jira_raw"}}
]
}
}connection_nameobjects[].tableobjects[].schemaobjects[].reportsource_urlconnector_optionsgdrive_optionsdatabricks bundle schemapipelines.ConnectorOptionstable_configurationscd_type: SCD_TYPE_2channel: PREVIEWingestion_definitionlibrariesdestination_table-troubleshoot| Symptom | Cause | Fix |
|---|---|---|
| The key the source reports isn't unique per row (NetSuite's line-level tables report | Set |
| A working connection fails auth | Refresh token revoked or expired | Reauthorize the connection in Catalog Explorer |
| Create fails or the connector isn't found | Previews toggle off, or | Check the page's Beta note and examples |
| Every update re-reads a whole table | No cursor (entity tables, formula fields, report snapshots) | Expected; Salesforce formula fields can be ingested incrementally (Beta) in a new pipeline, API only: Ingest formula fields incrementally |
| Too many tables | Per-pipeline cap on | Split the pipeline |