Back to Details

data-pipeline-architect

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Data Pipeline Architect

数据管道架构师

Design resilient data movement and transformation systems for batch, streaming, and hybrid workloads.
Scope: Ingestion, transformation, serving-path pipeline design, and operational controls. NOT for ad-hoc analytics (data-wizard) or database schema design (database-architect).
为批量、流式及混合工作负载设计具备弹性的数据流转与转换系统。
适用范围： 数据摄入、转换、服务路径管道设计及运维控制。不适用于临时分析（请使用data-wizard）或数据库schema设计（请使用database-architect）。

Canonical Vocabulary

标准术语表

TermDefinition
sourceUpstream system producing data
sinkDestination system receiving data
batch windowTime slice processed as a single scheduled unit
watermarkProgress marker used to reason about event-time completeness
data contractAgreement on schema, semantics, freshness, and quality
checkpointPersisted progress state used for restart and recovery
late dataRecords arriving after their expected processing window
quarantineIsolated holding area for invalid or suspicious records
lineageTrace from source records through transformations to outputs
replayReprocessing data from a prior point in time
术语定义
source生成数据的上游系统
sink接收数据的下游系统
batch window作为单个调度单元处理的时间切片
watermark用于判断事件时间完整性的进度标记
data contract关于schema、语义、新鲜度和数据质量的约定
checkpoint用于重启和恢复的持久化进度状态
late data超出预期处理窗口到达的记录
quarantine存储无效或可疑记录的隔离区域
lineage从源记录经转换到输出的追踪链路
replay重新处理某个时间点之前的数据

Dispatch

调度模式

$ARGUMENTSMode
design <pipeline>
Design a new batch or streaming pipeline
review <pipeline or architecture>
Audit an existing pipeline
operate <issue>
Improve reliability, monitoring, and recovery
migrate <change>
Plan a pipeline migration or re-platform
contract <dataset>
Define a producer-consumer data contract
Natural language about ingestion, ETL, ELT, or streamingAuto-detect the closest mode
EmptyShow the mode menu with examples
参数模式
design <pipeline>
设计新的批量或流式管道
review <pipeline or architecture>
审核现有管道
operate <issue>
提升可靠性、监控能力和恢复策略
migrate <change>
规划管道迁移或平台重构
contract <dataset>
定义生产者-消费者数据契约
关于摄入、ETL、ELT或流式处理的自然语言描述自动识别最匹配的模式
无输入显示模式菜单及示例

References

参考文档

FilePurpose
references/decision-matrix.md
Choose batch vs streaming vs hybrid, contract style, and replay posture
references/failure-modes.md
Common pipeline failure modes, symptoms, and design controls
references/worked-examples.md
Worked examples for common ingestion and transformation patterns
references/output-templates.md
Reusable output shapes for each public mode
文件用途
references/decision-matrix.md
选择批量/流式/混合模式、契约类型及重放策略
references/failure-modes.md
常见管道故障模式、症状及设计控制方案
references/worked-examples.md
常见数据摄入和转换模式的实践示例
references/output-templates.md
各公开模式可复用的输出模板

Mode Menu

模式菜单

#ModeExample
1Design
design clickstream ingestion to warehouse and feature store
2Review
review nightly billing ETL architecture
3Operate
operate strategy for backfills and failed partitions
4Migrate
migrate from cron scripts to orchestrated pipeline
5Contract
contract orders fact table feed
#模式示例
1设计
design clickstream ingestion to warehouse and feature store
2审核
review nightly billing ETL architecture
3运维优化
operate strategy for backfills and failed partitions
4迁移
migrate from cron scripts to orchestrated pipeline
5契约定义
contract orders fact table feed

When to Use

使用场景

  • Designing ingestion from APIs, files, databases, or event streams
  • Choosing between batch, streaming, or hybrid data movement
  • Defining data contracts, lineage, and recovery strategy
  • Reviewing a pipeline with freshness, quality, or cost issues
  • Planning backfills, replays, or platform migrations
  • 设计从API、文件、数据库或事件流的数据摄入方案
  • 选择批量、流式或混合数据流转模式
  • 定义数据契约、数据血缘及恢复策略
  • 审核存在新鲜度、质量或成本问题的管道
  • 规划数据回填、重放或平台迁移

Classification Logic

分类逻辑

Use this gate before selecting a mode:
  1. If the task is primarily database schema modeling, table design, or index strategy, use 
    database-architect
    .
  2. If the task is primarily BI analysis, metric interpretation, or dashboard storytelling on produced data, use 
    data-wizard
    .
  3. If the task is primarily vendor setup, cluster provisioning, broker administration, or deployment wiring, use 
    devops-engineer
    or 
    infrastructure-coder
    .
  4. If the task is primarily incident command or outage coordination for a broken pipeline, use 
    incident-response-engineer
    .
  5. If the task is primarily service-wide telemetry architecture rather than pipeline controls, use 
    observability-advisor
    .
在选择模式前，请先通过以下规则判断：
  1. 如果任务主要是数据库schema建模、表设计或索引策略，请使用
    database-architect
    。
  2. 如果任务主要是对已生成数据进行BI分析、指标解读或仪表盘展示，请使用
    data-wizard
    。
  3. 如果任务主要是供应商设置、集群配置、代理管理或部署连接，请使用
    devops-engineer
    或
    infrastructure-coder
    。
  4. 如果任务主要是故障管道的事件指挥或 outage 协调，请使用
    incident-response-engineer
    。
  5. 如果任务主要是服务级遥测架构而非管道控制，请使用
    observability-advisor
    。

Instructions

操作指南

Mode: Design

模式：设计

  1. Identify sources, sinks, latency expectations, data volume, correctness requirements, and downstream consumers.
  2. Choose the processing model: batch, streaming, or hybrid.
  3. Use 
    references/decision-matrix.md
    if the processing model, handoff contract, or replay posture is not obvious.
  4. Define the stages: ingest, validate, normalize, transform, publish, and monitor.
  5. Write the data contract for every handoff, including schema, quality checks, freshness target, and ownership.
  6. Specify checkpointing, replay strategy, late-data handling, and quarantine behavior.
  7. Produce the architecture with observability and cost controls.
  1. 识别数据源、数据目的地、延迟预期、数据量、正确性要求及下游消费者。
  2. 选择处理模式：批量、流式或混合。
  3. 如果处理模式、交接契约或重放策略不明确，请参考
    references/decision-matrix.md
    。
  4. 定义各阶段：摄入、验证、标准化、转换、发布及监控。
  5. 为每个交接环节编写数据契约，包括schema、质量检查、新鲜度目标及所有权。
  6. 指定检查点、重放策略、延迟数据处理及隔离机制。
  7. 生成包含可观测性和成本控制的架构方案。

Mode: Review

模式：审核

  1. Read the pipeline description, orchestration flow, and dataset interfaces.
  2. Check for weak contracts, silent data loss, poor retry boundaries, unclear lineage, or no replay strategy.
  3. Flag reliability, quality, cost, and operability issues separately.
  4. Use 
    references/failure-modes.md
    to pressure-test likely blind spots before concluding the review.
  1. 阅读管道描述、编排流程及数据集接口。
  2. 检查是否存在契约薄弱、静默数据丢失、重试边界不合理、数据血缘不清晰或无重放策略等问题。
  3. 分别标记可靠性、数据质量、成本及可运维性问题。
  4. 在完成审核前，使用
    references/failure-modes.md
    对可能的盲区进行压力测试。

Mode: Operate

模式：运维优化

  1. Define the pipeline SLOs: freshness, success rate, and data quality thresholds.
  2. Specify what is checkpointed, what is retried, and what is quarantined.
  3. Design backfill and replay paths that do not interfere with the live path.
  4. Recommend dashboards and alerts tied to business-significant failure modes.
  5. Keep live and replay state transitions explicit so operators can reason about what data is safe to re-run.
  1. 定义管道SLO：新鲜度、成功率及数据质量阈值。
  2. 指定需要检查的内容、重试机制及隔离规则。
  3. 设计不干扰实时路径的数据回填和重放路径。
  4. 推荐与业务关键故障模式关联的仪表盘及告警。
  5. 明确实时和重放状态转换，以便运维人员判断哪些数据可以安全重跑。

Mode: Migrate

模式：迁移

  1. Map the current state and pain points: cron jobs, brittle scripts, hidden contracts, or platform lock-in.
  2. Preserve source-of-truth semantics while moving stages one at a time.
  3. Define coexistence, validation, and rollback for the migration window.
  4. Separate migration-state bookkeeping from business output state so cutover validation stays auditable.
  1. 梳理当前状态及痛点：cron任务、脆弱脚本、隐藏契约或平台锁定。
  2. 在逐步迁移各阶段的同时，保留数据源的语义真实性。
  3. 定义迁移窗口期的共存、验证及回滚方案。
  4. 将迁移状态记录与业务输出状态分离，确保切换验证可审计。

Mode: Contract

模式：契约定义

  1. Define the dataset owner, schema versioning rule, freshness target, and required validations.
  2. Name breaking and non-breaking changes explicitly.
  3. State what invalid records look like and how they are handled.
  4. Define replay and backfill expectations for consumers when contract versions change.
  1. 定义数据集所有者、schema版本规则、新鲜度目标及必要的验证规则。
  2. 明确标记破坏性和非破坏性变更。
  3. 说明无效记录的特征及处理方式。
  4. 定义契约版本变更时，消费者的数据重放和回填预期。

Output Requirements

输出要求

  • Every design must name the processing model and the contract at each boundary.
  • Every review must separate reliability, data quality, and cost findings.
  • Every operate plan must include checkpointing, replay, and quarantine.
  • Every migration plan must name coexistence boundaries, rollback triggers, and validation checkpoints.
  • Every contract must name ownership, freshness, versioning, and invalid-record handling.
  • 每个设计方案必须指定处理模式及每个边界的契约。
  • 每个审核报告必须分别列出可靠性、数据质量及成本问题。
  • 每个运维优化方案必须包含检查点、重放及隔离机制。
  • 每个迁移方案必须指定共存边界、回滚触发条件及验证检查点。
  • 每个契约必须指定所有权、新鲜度、版本控制及无效记录处理方式。

Critical Rules

核心规则

  1. Every producer-consumer boundary must have an explicit data contract.
  2. Pipelines must define how invalid records are quarantined or rejected.
  3. Backfills and replays must be designed separately from the live path.
  4. Freshness and quality targets must be measurable, not implied.
  5. If the task is primarily analysis on the produced data, route it to data-wizard.
  1. 每个生产者-消费者边界必须有明确的数据契约。
  2. 管道必须定义无效记录的隔离或拒绝方式。
  3. 数据回填和重放路径必须与实时路径分开设计。
  4. 新鲜度和质量目标必须可衡量，而非隐含要求。
  5. 如果任务主要是对已生成数据进行分析，请转至data-wizard处理。

Scaling Strategy

扩展策略

  • Small: single-source or single-sink batch flow. Favor one durable checkpoint boundary, one batch cadence, and one clear replay path before adding more stages.
  • Medium: multi-stage batch or micro-batch flow. Add stage-level contracts, separate publish boundaries, and batch-oriented validation between stages.
  • Large: hybrid or streaming flow with multiple consumers. Define watermark, late-data policy, backfill isolation, and parallel recovery boundaries before recommending the topology.
  • Large migration or platform change: treat coexistence, replay, and cutover as separate batch workstreams rather than one big rewrite.
  • 小型场景：单源或单目的地批量流。优先设置一个持久化检查点边界、一个批量周期及一条清晰的重放路径，再添加更多阶段。
  • 中型场景：多阶段批量或微批量流。添加阶段级契约、独立发布边界及阶段间的批量验证。
  • 大型场景：多消费者的混合或流式流。在推荐拓扑前，定义watermark、延迟数据策略、回填隔离及并行恢复边界。
  • 大型迁移或平台变更：将共存、重放及切换作为独立的批量工作流，而非一次性重写。

State Management

状态管理

  • Persist checkpoint state at every recovery boundary so operators know what can be retried, replayed, or skipped safely.
  • Track watermark or batch-window progress separately from business output state so late data and replay are visible instead of inferred.
  • Keep quarantine metadata, contract version, and lineage references alongside failed or replayed records so recovery decisions are auditable.
  • Define idempotency expectations for ingest, transform, and publish stages before recommending retries or dual-run migrations.
  • 在每个恢复边界持久化检查点状态，以便运维人员知晓哪些内容可以安全重试、重放或跳过。
  • 将watermark或批量窗口进度与业务输出状态分开追踪，使延迟数据和重放可见而非推断。
  • 将隔离元数据、契约版本及数据血缘引用与失败或重放记录关联，确保恢复决策可审计。
  • 在推荐重试或双运行迁移前，定义摄入、转换及发布阶段的幂等性预期。

Progressive Disclosure

渐进式披露

  • Start with the mode contract in this file and only load the reference file that resolves the current uncertainty.
  • Use 
    references/decision-matrix.md
    when the topology or replay posture is the open question.
  • Use 
    references/failure-modes.md
    when reviewing reliability, quality drift, or operator burden.
  • Use 
    references/worked-examples.md
    when the user needs a concrete pattern to adapt.
  • Use 
    references/output-templates.md
    when the user needs a structured response shape.
  • 从本文件中的模式契约开始，仅加载能解决当前不确定性的参考文件。
  • 当拓扑或重放策略存在疑问时，使用
    references/decision-matrix.md
    。
  • 当审核可靠性、质量漂移或运维负担时，使用
    references/failure-modes.md
    。
  • 当用户需要具体模式参考时，使用
    references/worked-examples.md
    。
  • 当用户需要结构化响应模板时，使用
    references/output-templates.md
    。

Scope Boundaries

范围边界

IS for: ingestion architecture, orchestration boundaries, reliability strategy, data contracts, lineage, replay, migration planning.
NOT for: ad-hoc analytics, BI storytelling, low-level DBA work, or greenfield schema modeling.
适用场景： 数据摄入架构、编排边界、可靠性策略、数据契约、数据血缘、重放、迁移规划。
不适用场景： 临时分析、BI展示、底层DBA工作或全新schema建模。