Back to Details

recsys-pipeline-architect

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

recsys-pipeline-architect

recsys-pipeline-architect

A spec-and-scaffold skill for building composable recommendation, ranking, and feed pipelines. It encodes the six-stage pattern — Source → Hydrator → Filter → Scorer → Selector → SideEffect — popularized by xAI's open-sourced For You algorithm (Apache 2.0). This skill is an independent reimplementation of the pattern (MIT) — no code copied from the original.
Upstream: https://github.com/mturac/recsys-pipeline-architect
一款用于构建可组合推荐、排序和信息流管道的规范与脚手架技能。它实现了由xAI开源的For You算法（Apache 2.0协议）推广的六阶段模式——Source → Hydrator → Filter → Scorer → Selector → SideEffect。本技能是该模式的独立重实现（MIT协议）——未复制原项目的任何代码。
上游仓库：https://github.com/mturac/recsys-pipeline-architect

When to Use

适用场景

  • User wants to build any system that picks "the top K items for a user/context"
  • User asks "how should I rank X" or describes a feed/personalization problem
  • User has a scoring function and needs the pipeline plumbing around it
  • User wants to migrate from a single relevance score to multi-action prediction with tunable weights
  • User is wrapping an LLM/ML scorer and needs filters, hydrators, side-effects, and a runnable scaffold in their stack (TypeScript / Go / Python)
  • Triggers: "recommendation system", "feed algorithm", "ranking pipeline", "for you feed", "candidate pipeline", "content recommender", "pipeline architecture for recsys", "RAG retrieval reranker"
  • 用户希望构建任何需为“用户/上下文选择前K个项目”的系统
  • 用户询问“我该如何对X进行排序”或描述了信息流/个性化相关问题
  • 用户已有评分函数，需要围绕它搭建管道架构
  • 用户希望从单一相关性评分迁移至带可调权重的多动作预测
  • 用户正在封装LLM/ML评分器，需要在其技术栈（TypeScript / Go / Python）中添加过滤器、数据增强器、副作用处理及可运行的脚手架
  • 触发关键词："recommendation system"、"feed algorithm"、"ranking pipeline"、"for you feed"、"candidate pipeline"、"content recommender"、"pipeline architecture for recsys"、"RAG retrieval reranker"

When NOT to Use

不适用场景

  • Model architecture work (transformer design, two-tower retrieval, embedding training) — this skill is plumbing around the model, not the model itself
  • Pure ML training pipelines — the scoring function is the user's responsibility
  • Operating a deployed pipeline (monitoring, autoscaling) — out of scope
  • 模型架构工作（Transformer设计、双塔检索、嵌入训练）——本技能负责模型周边的架构搭建，而非模型本身
  • 纯ML训练管道——评分函数由用户自行负责
  • 已部署管道的运维（监控、自动扩缩容）——不在本技能范围内

The six-stage framework

六阶段框架

#StageJobParallel?
1SourceFetch candidates from one or more originsYes — multiple sources run in parallel
2HydratorEnrich each candidate with metadata needed for filtering and scoringYes — independent hydrators run in parallel
3FilterDrop candidates that should never be shown (blocked, expired, duplicate, ineligible)Sequential — each filter sees fewer items
4ScorerAssign each surviving candidate one or more scoresSequential — later scorers see earlier scores
5SelectorSort by final score, return top KSingle op
6SideEffectCache served IDs, log impressions, emit events, update countersAsync — must never block the response
序号阶段职责是否可并行？
1Source从一个或多个来源获取候选项目是——多来源可并行运行
2Hydrator为每个候选项目添加过滤和评分所需的元数据是——独立的数据增强器可并行运行
3Filter剔除不应展示的候选项目（已屏蔽、过期、重复、不符合资格）串行——每个过滤器处理的项目数量逐渐减少
4Scorer为每个留存的候选项目分配一个或多个评分串行——后续评分器可获取前期评分结果
5Selector根据最终评分排序，返回前K个项目单一操作
6SideEffect缓存已展示的ID、记录曝光数据、触发事件、更新计数器异步——绝不能阻塞响应

Why this exact order

为何采用此顺序

  • Sources before hydration: know what candidates exist before paying to enrich them
  • Hydration before filtering: many filters need metadata the source did not provide
  • Filtering before scoring: scoring is the expensive stage; drop the ineligible first
  • Scorer chain (not single scorer): real systems compose ML scoring + diversity reranking + business rules
  • Selector after scoring: keeps scoring deterministic and cacheable
  • SideEffects last and async: side effects must never block the user response
  • 先获取来源再做数据增强：先明确候选项目范围，再为其添加元数据，避免资源浪费
  • 先数据增强再过滤：许多过滤器需要来源未提供的元数据
  • 先过滤再评分：评分是高成本阶段，先剔除不符合资格的项目
  • 评分链（而非单一评分器）：实际系统需结合ML评分、多样性重排序、业务规则
  • 评分后再选择：保证评分的确定性和可缓存性
  • 最后异步处理副作用：副作用绝不能阻塞用户响应

Workflow when invoked

调用后的工作流程

Walk the user through these eight steps:
  1. Clarify the use case (one round, three questions): items being ranked? input context? language/runtime?
  2. Identify the candidate sources: usually in-network (followed/owned/subscribed) + out-of-network (ML retrieval / trending / similar-to-liked)
  3. List required hydrations: for each filter and scorer, what data does it need that the source did not provide?
  4. List the filters: duplicate, self, age, block/mute, previously-served, eligibility. Order matters — cheap before expensive.
  5. Design the scorer chain: primary (ML) → combiner (multi-action with weights) → diversity → business rules
  6. Selector: sort descending by final score, take top K (or stratified mix for in-network/out-of-network)
  7. SideEffects: cache served IDs, emit impression events, update counters, log analytics — all fire-and-forget
  8. Generate the scaffold in the user's stack
引导用户完成以下八个步骤：
  1. 明确使用场景（一轮沟通，三个问题）：待排序的项目类型？输入上下文？使用的语言/运行时？
  2. 确定候选项目来源：通常包括网络内（已关注/拥有/订阅的内容）+ 网络外（ML检索/热门/相似内容）
  3. 列出所需的数据增强项：针对每个过滤器和评分器，来源未提供但必需的数据是什么？
  4. 列出过滤器：重复项、自身内容、时效、屏蔽/静音、已展示过的内容、资格验证。顺序很重要——低成本过滤器优先于高成本过滤器。
  5. 设计评分链：主评分（ML）→ 组合器（带权重的多动作）→ 多样性调整 → 业务规则
  6. 选择器：按最终评分降序排序，取前K个（或按网络内/外分层混合）
  7. 副作用处理：缓存已展示ID、触发曝光事件、更新计数器、记录分析数据——全部采用“触发即遗忘”模式
  8. 生成用户技术栈对应的脚手架

Key trade-offs to surface (don't default silently)

需要明确的关键权衡（不可默认选择）

1. Single score vs multi-action prediction

1. 单一评分 vs 多动作预测

  • Single score: train one model to predict relevance. To change behavior → retrain.
  • Multi-action: predict 
    P(action)
    for many actions (read, like, share, skip, report), combine with weights at serving time. To change behavior → change weights. No retraining.
The X For You system uses multi-action with both positive and negative weights. Recommend multi-action when the user expects to tune frequently.
  • 单一评分：训练一个模型预测相关性。如需调整行为→重新训练模型。
  • 多动作预测：预测多个动作的
    P(action)
    （阅读、点赞、分享、跳过、举报），在服务阶段结合权重计算。如需调整行为→修改权重。无需重新训练。
X的For You系统采用带正负权重的多动作预测。当用户需要频繁调优时，推荐使用多动作预测。

2. Candidate isolation in scoring

2. 评分时的候选项目隔离

  • Isolated: each candidate scored independently. Deterministic, cacheable.
  • Joint: candidates attend to each other during scoring (e.g., transformer over batch). More expressive but non-deterministic across batches.
Default to isolation. Joint only when there's a specific reason (e.g., explicit batch-aware diversity).
  • 隔离式：每个候选项目独立评分。具有确定性、可缓存性。
  • 联合式：评分时候选项目相互影响（如对批次使用Transformer）。表达能力更强，但批次间结果非确定性。
默认选择隔离式。仅当有特定需求时（如明确的批次感知多样性调整）才使用联合式。

3. Online vs offline

3. 在线 vs 离线

  • Request-time (online): pipeline runs on each request. Latency budget: 100–300ms. Default.
  • Pre-computed (offline batch): pipeline runs periodically, results cached. Lower latency, lower freshness.
  • Hybrid: candidate retrieval offline, ranking online.
  • 请求时（在线）：管道在每次请求时运行。延迟预算：100–300ms。默认选择。
  • 预计算（离线批次）：管道定期运行，结果缓存。延迟更低，但新鲜度不足。
  • 混合式：候选项目检索离线完成，排序在线进行。

Hard rules

硬性规则

  1. Do not invent benchmark numbers. "How much faster?" → "depends on workload, run it yourself."
  2. Attribution discipline. When the pattern is referenced, attribute as "popularized by xAI's open-sourced For You algorithm" / 
    github.com/xai-org/x-algorithm
    (Apache 2.0).
  3. No trademark use. Do not name the user's artifact "X-like" or use "For You" branding. Pattern is free; brand is not. Suggested naming: "candidate pipeline", "feed pipeline", "ranking pipeline", "recsys pipeline".
  4. Surface trade-offs. Multi-action vs single, isolation vs joint, online vs offline — never default silently.
  5. The generated scaffold must run. No pseudocode passing as code.
  6. Filter order matters. Cheap before expensive. Universal before user-specific.
  7. Side effects never block. Wrap in fire-and-forget patterns (goroutines / promises without await / asyncio tasks).
  1. 不得编造基准数据。若被问“速度提升多少？”→回答“取决于工作负载，请自行测试。”
  2. 规范归属标注。当引用该模式时，标注为“由xAI开源的For You算法推广” / 
    github.com/xai-org/x-algorithm
    （Apache 2.0协议）。
  3. 不得使用商标。不得将用户的产物命名为“类X”或使用“For You”品牌。模式可免费使用；品牌不可。建议命名：“候选项目管道”、“信息流管道”、“排序管道”、“推荐系统管道”。
  4. 明确权衡选项。多动作vs单一、隔离vs联合、在线vs离线——绝不能默认选择而不告知用户。
  5. 生成的脚手架必须可运行。不得用伪代码替代可执行代码。
  6. 过滤器顺序至关重要。低成本过滤器优先于高成本过滤器。通用过滤器优先于用户特定过滤器。
  7. 副作用绝不能阻塞。采用“触发即遗忘”模式（goroutines / 无需await的promise / asyncio任务）。

Anti-Patterns

反模式

  • Scoring before filtering (wastes compute on candidates that will be dropped anyway)
  • Synchronous side effects (cache writes / impression emits blocking the response)
  • A single "relevance" score when the product needs to tune for multiple objectives (engagement vs safety vs diversity vs ads)
  • Joint scoring as default (non-deterministic, harder to cache, doesn't compose with reranking stages)
  • Generating pseudocode "for illustration" — the scaffold must actually run
  • 先评分再过滤（浪费计算资源在最终会被剔除的候选项目上）
  • 同步处理副作用（缓存写入/曝光事件触发阻塞响应）
  • 当产品需要针对多目标调优（参与度vs安全性vs多样性vs广告）时，仅使用单一“相关性”评分
  • 默认使用联合评分（非确定性、难以缓存、无法与重排序阶段组合）
  • 生成“用于演示”的伪代码——脚手架必须实际可运行

Upstream contents

上游仓库内容

The upstream repository at https://github.com/mturac/recsys-pipeline-architect ships:
  • Full 
    SKILL.md
    with the complete 8-step workflow
  • 5 load-on-demand reference docs: interfaces in 4 languages (TS/Go/Python/Rust), multi-action scoring pattern, candidate isolation, filter cookbook (12 patterns), scorer cookbook (weighted sum, MMR, diversity penalty, position debiasing)
  • 3 runnable example scaffolds, every one green on its test suite:
    • Strapi v5 plugin (TypeScript / Jest — 3/3 pass)
    • Zentra-compatible pipeline (Go with generics — 3/3 pass)
    • PMAI task prioritizer (Python / FastAPI / pytest — 3/3 pass)
  • v0.1.0 release tagged
  • MIT license; pattern attributed to xAI X For You algorithm (Apache 2.0)
Install via skills.sh: 
npx skills add mturac/recsys-pipeline-architect
上游仓库https://github.com/mturac/recsys-pipeline-architect包含：
  • 完整的
    SKILL.md
    ，包含全部8步工作流程
  • 5个按需加载的参考文档：4种语言（TS/Go/Python/Rust）的接口定义、多动作评分模式、候选项目隔离、过滤器手册（12种模式）、评分器手册（加权求和、MMR、多样性惩罚、位置偏差校正）
  • 3个可运行的示例脚手架，全部测试用例通过：
    • Strapi v5插件（TypeScript / Jest — 3/3通过）
    • Zentra兼容管道（带泛型的Go — 3/3通过）
    • PMAI任务优先级分配器（Python / FastAPI / pytest — 3/3通过）
  • 已标记v0.1.0版本
  • MIT协议；模式归属标注为xAI X For You算法（Apache 2.0协议）
通过skills.sh安装：
npx skills add mturac/recsys-pipeline-architect