When data must flow through a fixed sequence of discrete transformations, such as in ETL jobs, streaming analytics, or CI/CD pipelines.
When reusing individual processing stages is needed, either independently or to scale bottleneck stages separately from others.
When failure isolation between stages is a critical requirement.
当数据必须经过一系列固定的离散转换步骤时，例如在ETL作业、流分析或CI/CD管道中。
当需要复用单个处理阶段时，无论是独立复用，还是单独扩展瓶颈阶段。
当各阶段之间的故障隔离是关键需求时。
Adoption Steps
实施步骤
Define Filters: Design each stage (filter) to perform a single, well-defined transformation. Each filter must have a clear input and output data schema.
Connect via Pipes: Connect the filters using "pipes," which can be implemented as streams, message queues, or in-memory channels. validate these pipes support back-pressure and buffering.
Maintain Stateless Filters: Where possible, design filters to be stateless. Any required state should be persisted externally or managed at the boundaries of the pipeline.
Instrument Each Stage: Implement monitoring for each filter to track key metrics such as latency, throughput, and error rates.
Orchestrate Deployments: Design the deployment strategy to allow each stage to be scaled horizontally and upgraded independently.
An Architecture Decision Record (ADR) documenting the filters, the chosen pipe technology, the error-handling strategy, and the tools for replaying data.
A suite of contract tests for each filter, plus integration tests that cover representative end-to-end pipeline executions.
Observability dashboards that visualize stage-level Key Performance Indicators (KPIs).
一份架构决策记录（ADR），记录过滤器、所选管道技术、错误处理策略以及数据重放工具。
针对每个过滤器的一套契约测试，以及覆盖典型端到端管道执行的集成测试。
可观测性仪表板，用于可视化各阶段的关键性能指标（KPI）。
Risks & Mitigations
风险与缓解措施
Single-Stage Bottlenecks:
Mitigation: Implement auto-scaling for individual filters. If a single filter remains a bottleneck, consider refactoring it into a more granular sub-pipeline.
Schema Drift Between Stages:
Mitigation: Centralize schema definitions in a shared repository and enforce compatibility tests as part of the CI/CD process to prevent breaking changes.
Back-Pressure Failures:
Mitigation: Conduct rigorous load testing to simulate high-volume scenarios. Validate that buffering, retry logic, and back-pressure mechanisms behave as expected under stress.