databricks-spark-structured-streaming
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseSpark Structured Streaming
Spark Structured Streaming
Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.
生产就绪的Spark Structured Streaming流处理管道。本技能提供详细模式与最佳实践的导航指引。
Quick Start
Quick Start
python
from pyspark.sql.functions import col, from_jsonpython
from pyspark.sql.functions import col, from_jsonBasic Kafka to Delta streaming
Basic Kafka to Delta streaming
df = (spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "broker:9092")
.option("subscribe", "topic")
.load()
.select(from_json(col("value").cast("string"), schema).alias("data"))
.select("data.*")
)
df.writeStream
.format("delta")
.outputMode("append")
.option("checkpointLocation", "/Volumes/catalog/checkpoints/stream")
.trigger(processingTime="30 seconds")
.start("/delta/target_table")
.format("delta")
.outputMode("append")
.option("checkpointLocation", "/Volumes/catalog/checkpoints/stream")
.trigger(processingTime="30 seconds")
.start("/delta/target_table")
undefineddf = (spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "broker:9092")
.option("subscribe", "topic")
.load()
.select(from_json(col("value").cast("string"), schema).alias("data"))
.select("data.*")
)
df.writeStream
.format("delta")
.outputMode("append")
.option("checkpointLocation", "/Volumes/catalog/checkpoints/stream")
.trigger(processingTime="30 seconds")
.start("/delta/target_table")
.format("delta")
.outputMode("append")
.option("checkpointLocation", "/Volumes/catalog/checkpoints/stream")
.trigger(processingTime="30 seconds")
.start("/delta/target_table")
undefinedCore Patterns
Core Patterns
| Pattern | Description | Reference |
|---|---|---|
| Kafka Streaming | Kafka to Delta, Kafka to Kafka, Real-Time Mode | See references/kafka-streaming.md |
| Real-Time Mode (RTM) | Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), | See references/real-time-mode.md |
| Lakebase Sink | Write streaming records into Lakebase Postgres with transactional upserts. Native | See references/lakebase-sink-python.md |
| Stream Joins | Stream-stream joins, stream-static joins | See references/stream-stream-joins.md, references/stream-static-joins.md |
| Multi-Sink Writes | Write to multiple tables, parallel merges | See references/multi-sink-writes.md |
| Merge Operations | MERGE performance, parallel merges, optimizations | See references/merge-operations.md |
| 模式 | 描述 | 参考 |
|---|---|---|
| Kafka流处理 | Kafka到Delta、Kafka到Kafka、实时模式 | 查看 references/kafka-streaming.md |
| 实时模式（RTM） | 亚秒级端到端延迟 —— 集群设置、槽位计算、支持的操作（包括DBR 18+版本的流-流内连接）、 | 查看 references/real-time-mode.md |
| Lakebase输出端 | 将流处理记录以事务性更新写入Lakebase Postgres。原生 | 查看 references/lakebase-sink-python.md |
| 流连接 | 流-流连接、流-静态连接 | 查看 references/stream-stream-joins.md、references/stream-static-joins.md |
| 多输出端写入 | 写入多张表、并行合并 | 查看 references/multi-sink-writes.md |
| 合并操作 | MERGE性能、并行合并、优化 | 查看 references/merge-operations.md |
Configuration
Configuration
| Topic | Description | Reference |
|---|---|---|
| Checkpoints | Checkpoint management and best practices | See references/checkpoint-best-practices.md |
| Stateful Operations | Watermarks, state stores, RocksDB configuration | See references/stateful-operations.md |
| Trigger & Cost | Trigger selection, cost optimization, RTM | See references/trigger-and-cost-optimization.md |
| 主题 | 描述 | 参考 |
|---|---|---|
| 检查点 | 检查点管理与最佳实践 | 查看 references/checkpoint-best-practices.md |
| 有状态操作 | 水印、状态存储、RocksDB配置 | 查看 references/stateful-operations.md |
| 触发器与成本 | 触发器选择、成本优化、RTM | 查看 references/trigger-and-cost-optimization.md |
Best Practices
Best Practices
| Topic | Description | Reference |
|---|---|---|
| Production Checklist | Comprehensive best practices | See references/streaming-best-practices.md |
| 主题 | 描述 | 参考 |
|---|---|---|
| 生产环境检查清单 | 全面的最佳实践 | 查看 references/streaming-best-practices.md |
Production Checklist
Production Checklist
- Checkpoint location is persistent (UC volumes, not DBFS)
- Unique checkpoint per stream
- Fixed-size cluster (no autoscaling for streaming)
- Monitoring configured (input rate, lag, batch duration)
- Exactly-once verified (txnVersion/txnAppId)
- Watermark configured for stateful operations
- Left joins for stream-static (not inner)
- 检查点位置为持久化存储（UC卷，而非DBFS）
- 每个流处理任务使用唯一的检查点
- 固定大小集群（流处理不使用自动扩缩容）
- 已配置监控（输入速率、延迟、批处理时长）
- 已验证Exactly-once语义（txnVersion/txnAppId）
- 为有状态操作配置水印
- 流-静态连接使用左连接（而非内连接）