task-scheduling

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Task Scheduling

任务调度

Decide when work runs and which worker runs it: fire jobs on a schedule (cron/delayed/recurring), hand each job to exactly one worker via a lease, and make sure it completes once despite crashes and retries. This sits on top of

messaging-streaming

queues — the queue is the transport; this skill adds the scheduling, leasing, priorities, and task-level idempotency. Getting it wrong shows up as jobs that never run, run twice (double charge, double email), or pile up until a worker fleet falls permanently behind.

决定工作何时执行以及由哪个Worker执行：按计划触发作业（Cron/延迟/周期性），通过租赁机制将每个作业分配给唯一的Worker，并确保即便发生崩溃和重试，作业也仅完成一次。本技能构建于

messaging-streaming

队列之上——队列作为传输层，本技能则增加了调度、租赁、优先级和任务级幂等性。若设计不当，会出现作业永不执行、重复执行（如重复扣费、重复发送邮件），或堆积至Worker集群彻底无法跟上进度的情况。

When to reach for this

适用场景

Work must run later (send a reminder in 24h), on a schedule (nightly rollups, hourly cron), or repeatedly (poll every 5 min); a slow operation is already off the request path (→

messaging-streaming

) and now needs reliable allocation to a pool of workers; jobs need priorities (paid before free) or fairness (no single tenant starves others); or a job must complete exactly once even though the worker holding it can crash mid-flight.

工作需要延迟执行（24小时后发送提醒）、按计划执行（夜间汇总、每小时Cron任务）或重复执行（每5分钟轮询一次）；耗时操作已脱离请求路径（→

messaging-streaming

），现在需要可靠分配给Worker池；作业需要优先级（付费用户任务优先于免费用户）或公平性（避免单个租户占用全部资源）；或者即便持有作业的Worker中途崩溃，作业也必须精确执行一次。

When NOT to

不适用场景

The caller needs the result inline — that's a synchronous call, not a scheduled job. A single fire-and-forget async step with no schedule, priority, or exactly-once need — a plain queue + idempotent consumer (

messaging-streaming

) is simpler; don't add a scheduler on top. One periodic job on one box — OS

cron

is fine until you have multiple schedulers or need history and retries. A long-running multi-step saga with rollback — reach for a durable workflow engine instead of hand-rolling state across jobs. Don't stand up Airflow/Celery "because we'll have batch jobs eventually" (YAGNI): it's a stateful control plane to operate and monitor.

调用方需要即时获取结果——这属于同步调用，而非定时作业。仅需单次即发即弃的异步步骤，无需调度、优先级或精确一次执行的需求——使用普通队列+幂等消费者（

messaging-streaming

）即可，无需额外添加调度器。单台机器上的单个周期性任务——在需要多调度器或历史记录、重试机制前，OS

cron

已足够。需要回滚的长期多步骤流程——应使用持久化工作流引擎，而非通过作业手动维护状态。不要因为“以后可能会有批处理作业”就搭建Airflow/Celery（YAGNI原则）：这是一个需要运维和监控的有状态控制平面。

Clarify first

先明确以下问题

Trigger type — scheduled (cron/at a time), delayed (run after N seconds), recurring (every N), or event-driven (a queue message arrives)? This decides whether a scheduler is even in scope.
Exactly-once vs at-least-once — is a duplicate run harmful (money, email) or harmless (idempotent recompute)? Drives the leasing + dedup design.
Latency budget vs throughput — must a delayed job fire within seconds of its time, or is "within a few minutes" fine? Tight timing is more expensive.
Priority / fairness — do some jobs jump the line, and must one tenant or job class be prevented from starving the rest? (→
```
back-of-the-envelope
```
for arrival vs. service rate.)
Job duration & variance — seconds or hours? Sets the visibility-timeout / lease length and whether long jobs need heartbeats.
Idempotency key — what identifies a task as the same task on retry? (The key contract is owned by
```
api-design
```
.)

触发类型——定时（Cron/指定时间）、延迟（N秒后执行）、周期性（每隔N时间）还是事件驱动（队列消息到达）？这决定是否需要调度器。
精确一次 vs 至少一次——重复执行是否有害（如扣费、发送邮件）或无害（幂等重计算）？这将驱动租赁+去重设计。
延迟预算 vs 吞吐量——延迟作业必须在指定时间的几秒内触发，还是“几分钟内”即可？严格的时间要求成本更高。
优先级/公平性——部分作业是否需要插队，是否需要防止单个租户或作业类型占用全部资源？（→ 参考
```
back-of-the-envelope
```
中的到达率 vs 服务率。）
作业时长与波动——几秒还是几小时？这决定了可见性超时/租赁时长，以及长作业是否需要心跳机制。
幂等键——重试时，如何识别同一任务？（键的定义由
```
api-design
```
负责。）

The options

可选方案

Scheduling trigger

OS cron / single scheduler — one process fires jobs on a crontab. Use when one node, a handful of jobs, no HA requirement.
Distributed scheduler (HA cron) — a leader-elected scheduler enqueues due jobs into a queue; followers stand by. Use when the schedule must survive a node loss and must not double-fire.
Delay queue / timer — jobs carry a "not before" time; the queue holds them until due (delivery delay, sorted-set scoring, or a timer wheel). Use when per-job delays vary and you don't want a cron tick.
Workflow/orchestration DAG — declared task dependencies with backfill and history (Airflow-style). Use when batch pipelines have dependencies and you need a run history and reruns.

Worker allocation

Pull (worker leasing) — workers poll the queue, lease a job for a visibility timeout, and ack/delete on success. Use when you want back-pressure for free and elastic, self-balancing workers. The default.
Push (dispatcher assigns) — a coordinator routes jobs to specific workers. Use when affinity/locality matters (a job must run where its data is).

Priority & fairness

Priority queues — separate high/low queues drained in order. Use when some classes must run first.
Weighted / fair scheduling — round-robin or weighted draw across per-tenant queues. Use when one tenant's burst must not starve others.

调度触发方式

OS cron / 单调度器——单个进程根据crontab触发作业。适用场景：单节点、少量作业、无高可用要求。
分布式调度器（高可用Cron）——通过选主机制确定的调度器将到期作业加入队列；备用调度器待命。适用场景：调度需在节点故障时仍能运行，且不会重复触发作业。
延迟队列/定时器——作业携带“最早执行时间”；队列将作业保留至到期（交付延迟、有序集合计分或时间轮机制）。适用场景：每个作业的延迟时间不同，且无需Cron触发。
工作流/编排DAG——声明任务依赖，支持回填和历史记录（Airflow风格）。适用场景：批处理流水线存在依赖，且需要运行历史和重跑功能。

Worker分配方式

拉取（Worker租赁）——Worker轮询队列，租赁作业并获得可见性超时时间，成功完成后确认/删除作业。适用场景：需要自动背压、弹性自平衡的Worker。这是默认方案。
推送（调度器分配）——协调器将作业路由至特定Worker。适用场景：存在亲和性/本地性要求（作业必须在数据所在节点运行）。

优先级与公平性

优先级队列——分离高/低优先级队列，按顺序消费。适用场景：部分类型的作业必须优先执行。
加权/公平调度——在租户队列间进行轮询或加权抽取。适用场景：单个租户的突发流量不能占用全部资源。

Trade-offs

权衡对比

Option	What it solves	What it worsens	Change it when
OS cron / single scheduler	Trivial; zero infra	SPOF — node dies, schedule stops; no retry/history	You need HA or missed-run recovery → distributed scheduler
Distributed scheduler (HA cron)	Survives node loss; no double-fire (leader-elected)	Needs leader election (→ `consistency-coordination` ); more moving parts	One box and one job is enough → OS cron
Delay queue / timer	Per-job delays without a cron tick; precise-ish timing	Far-future jobs sit in the queue; timer accuracy bounded by poll interval	Delays are uniform/periodic → cron; dependencies exist → DAG
Workflow DAG (Airflow-style)	Dependencies, backfill, run history, reruns	Heavy control plane; scheduler latency; overkill for single jobs	Jobs are independent one-shots → plain queue + scheduler
Pull (worker leasing)	Self-balancing, elastic, natural back-pressure	At-least-once: lease expiry on a slow job re-runs it (need idempotency)	A job must run on a specific node (data locality) → push
Push (dispatcher)	Affinity/locality; central control	Dispatcher is a bottleneck/SPOF; must track worker health	No locality need → pull is simpler
Priority queues	Important work runs first	Low-priority starvation under sustained load	Fairness across tenants matters → weighted/fair
Weighted / fair scheduling	No tenant starves another	More complex; per-tenant accounting	Only one workload class exists → single queue

方案	解决的问题	带来的问题	何时更换
OS cron / 单调度器	实现简单；无需额外基础设施	单点故障——节点宕机则调度停止；无重试/历史记录	需要高可用或故障恢复→分布式调度器
分布式调度器（高可用Cron）	节点故障时仍能运行；不会重复触发作业（通过选主机制）	需要选主机制（→ `consistency-coordination` ）；组件更多	仅需单节点单作业→OS cron
延迟队列/定时器	无需Cron触发即可实现单作业延迟； timing相对精准	远期作业会占用队列资源；定时器精度受轮询间隔限制	延迟时间统一/周期性→Cron；存在依赖→DAG
工作流DAG（Airflow风格）	支持依赖、回填、运行历史、重跑	控制平面繁重；调度延迟高；单作业场景过于冗余	作业为独立一次性任务→普通队列+调度器
拉取（Worker租赁）	自平衡、弹性、天然背压	至少一次执行：慢作业租赁超时会导致重新执行（需要幂等性）	作业必须在特定节点运行（数据本地性）→推送
推送（调度器）	支持亲和性/本地性；集中控制	调度器是瓶颈/单点故障；必须跟踪Worker健康状态	无本地性需求→拉取更简单
优先级队列	重要作业优先执行	持续高负载下低优先级作业会被饿死	需要租户间公平性→加权/公平调度
加权/公平调度	避免租户资源占用不均	实现更复杂；需要按租户统计	仅有一种工作负载类型→单队列

Behavior under stress

压力下的表现

A scheduler can quietly fall behind, or it can amplify an outage by re-dispatching work a struggling fleet can't finish.

Backlog growth: arrival rate exceeds worker throughput; due jobs queue up and "scheduled for 09:00" runs at 09:40. End-to-end delay climbs while CPU looks fine. Mitigate: alarm on oldest-due-job age and queue depth, not just rate; scale workers; shed or defer low-priority jobs.
Lease expiry / re-run storm (the classic): a job runs longer than its visibility timeout, the lease expires, the queue redelivers it to a second worker, and now two workers run it — wasting capacity and, without idempotency, double-applying side effects. Mitigate: set the timeout above p99 job duration, heartbeat to extend the lease on long jobs, and make tasks idempotent/deduped.
Poison task: a job that always fails is retried forever, burning workers and re-loading a sick downstream. Mitigate: cap retries with backoff+jitter, then route to a dead-letter queue (retries/backoff/DLQ are owned by
```
resilience-failure
```
).
Thundering herd at the tick: thousands of cron jobs all scheduled at the top of the hour fire at once and stampede a downstream. Mitigate: jitter the schedule, spread triggers, or rate-limit dispatch.
Scheduler split-brain: two schedulers both think they're leader and double-enqueue every recurring job. Mitigate: a single leader via leader election + fencing (→
```
consistency-coordination
```
); idempotent enqueue keyed by (job, scheduled_time).
Starvation: a flood of high-priority or one noisy tenant's jobs starves everyone else. Mitigate: fair/weighted scheduling and per-tenant concurrency caps.

Monitor: oldest-due-job age (the best lateness signal), queue depth per priority, lease-expiry / redelivery rate, retry and DLQ rate, worker utilization, and per-tenant share.

调度器可能会悄悄落后进度，或通过重新分配作业来放大故障，导致Worker集群无法处理。

积压增长：到达率超过Worker吞吐量；到期作业堆积，“计划09:00执行”的作业实际在09:40运行。端到端延迟上升，但CPU使用率看似正常。缓解措施：监控最早到期作业的等待时长和队列深度，而非仅监控速率；扩容Worker；丢弃或延迟低优先级作业。
租赁超时/重跑风暴（经典问题）：作业运行时长超过可见性超时，租赁到期，队列将作业重新分发给第二个Worker，导致两个Worker同时运行该作业——浪费资源，且若无幂等性会产生重复副作用。缓解措施：将超时时间设置为超过p99作业时长，长作业通过心跳机制延长租赁时间，确保任务幂等/可去重。
有毒任务：始终失败的作业被无限重试，消耗Worker资源并持续压垮下游服务。缓解措施：设置重试次数上限并配合退避+抖动，之后将任务路由至死信队列（重试/退避/DLQ由
```
resilience-failure
```
负责）。
整点突发流量：数千个Cron任务都在整点触发，导致下游服务被冲击。缓解措施：给调度时间添加抖动，分散触发，或限制分发速率。
调度器脑裂：两个调度器都认为自己是主节点，导致每个周期性作业被重复加入队列。缓解措施：通过选主机制+ fencing（→
```
consistency-coordination
```
）确保只有一个主节点；按（作业，调度时间）作为键实现幂等入队。
饥饿问题：大量高优先级作业或单个租户的突发流量占用全部资源，导致其他作业无法执行。缓解措施：使用公平/加权调度，设置租户并发上限。

监控指标：最早到期作业的等待时长（最佳延迟信号）、各优先级队列深度、租赁超时/重发率、重试和DLQ率、Worker利用率、租户资源占比。

How to apply

实施步骤

Clarify the inputs. Settle trigger type, exactly-once vs at-least-once, latency budget, priority/fairness, and job duration (see Clarify first). If the work is a single async step with no schedule or priority, stop — a plain queue + idempotent consumer (
```
messaging-streaming
```
) is enough.
Pick from the trade-off table. Choose a trigger (cron → distributed scheduler → delay queue → DAG, cheapest that fits), an allocation model (pull leasing is the default; push only for locality), and a priority/fairness model only if more than one class exists.
Set the key knobs. Visibility timeout above p99 job duration, retry cap + backoff before the DLQ, the dedup/idempotency key per task, the lease heartbeat interval for long jobs, and per-tenant concurrency limits.
Stress-test the choice. Walk backlog growth, lease-expiry re-runs, poison tasks, tick stampede, scheduler split-brain, and starvation. Confirm a mitigation exists for each one the workload can trigger.
Size it with numbers. Workers needed = arrival rate × avg job seconds / target concurrency (Little's law); confirm sustained throughput drains peak arrival, and that far-future delayed jobs fit storage (→ Numbers that matter).
Pick a provider. Default to the generic recipe; only open a provider file if the user named a cloud (see Choosing a provider).

明确输入条件：确定触发类型、精确一次/至少一次、延迟预算、优先级/公平性、作业时长（见「先明确以下问题」）。若仅为单个无调度或优先级需求的异步步骤，则无需继续——普通队列+幂等消费者（
```
messaging-streaming
```
）已足够。
根据权衡表选型：选择触发方式（Cron→分布式调度器→延迟队列→DAG，选择满足需求的最精简方案）、分配模型（默认拉取租赁；仅在需要本地性时选择推送），仅当存在多种作业类型时才选择优先级/公平性模型。
设置关键参数：可见性超时超过p99作业时长，重试次数上限+退避后进入DLQ，每个任务的去重/幂等键，长作业的心跳间隔，租户并发上限。
压力测试选型：模拟积压增长、租赁超时重跑、有毒任务、整点突发流量、调度器脑裂、饥饿问题等场景，确认每种场景都有对应的缓解措施。
量化规模：所需Worker数量=到达率×平均作业时长/目标并发数（利特尔法则）；确认持续吞吐量能处理峰值到达率，且远期延迟作业的存储需求满足（→ 关键数值）。
选择服务商：默认使用通用方案；仅当用户指定云服务商时，查看
```
references/providers/<provider>.md
```
中的托管服务映射、配额/限制和服务商特定权衡。若该服务商无对应文档，则使用通用方案。

Dos and don'ts

注意事项

Default to pull-based worker leasing with a visibility timeout; it self-balances.
Set the visibility timeout above p99 job duration and heartbeat-extend long jobs.
Make every task idempotent (dedup on a task key) so a re-run is harmless.
Elect a single leader for the scheduler and key recurring enqueues by (job, time).
Cap retries with backoff+jitter, then dead-letter; alarm on oldest-due-job age.

Don't

Don't assume a leased job ran once — at-least-once means design for duplicates.
Don't schedule every cron job at :00 — jitter so the tick doesn't stampede.
Don't run two schedulers without leader election (split-brain double-fires).
Don't retry a poison task forever; cap it and dead-letter it.
Don't reach for Airflow/Celery before a schedule, priority, or HA need is real (YAGNI).

建议

默认使用基于拉取的Worker租赁+可见性超时；该方案可自平衡。
将可见性超时设置为超过p99作业时长，长作业通过心跳延长租赁时间。
确保每个任务都是幂等的（通过任务键去重），使重跑无副作用。
为调度器选主，并按（作业，时间）作为周期性入队的键。
设置重试次数上限+退避+抖动，之后进入死信队列；监控最早到期作业的等待时长。

禁止

不要假设租赁的作业仅执行一次——至少一次执行意味着要为重复执行做设计。
不要将所有Cron任务都设置在整点触发——添加抖动避免突发流量。
不要在无选主机制的情况下运行多个调度器（脑裂会导致重复触发）。
不要无限重试有毒任务；设置上限后转入死信队列。
不要在实际需要调度、优先级或高可用前就搭建Airflow/Celery（YAGNI原则）。

Numbers that matter

关键数值

Size the worker pool with Little's law: in-flight jobs = arrival rate × average job duration, so workers ≈ peak arrival × avg seconds-per-job / per-worker concurrency. Sustained drain must exceed peak arrival or the backlog never clears. Set the visibility timeout above the p99 job duration (a too-short timeout is the #1 cause of duplicate runs); set far-future delay storage = delayed-job rate × max delay × job size. Don't restate the latency/QPS tables — pull the rates and durations from

back-of-the-envelope

使用利特尔法则计算Worker池规模：在运行作业数=到达率×平均作业时长，因此Worker数量≈峰值到达率×平均作业秒数/单Worker并发数。持续吞吐量必须超过峰值到达率，否则积压永远无法清除。可见性超时设置为超过p99作业时长（过短的超时是重复执行的头号原因）；远期延迟作业存储需求=延迟作业率×最大延迟×作业大小。无需重复延迟/QPS表格——从

back-of-the-envelope

获取速率和时长数据。

Interface sketch

接口示例

A scheduled task is a contract. Define it, not "a job":

Task envelope: stable
```
task_id
```
/ idempotency key (dedup on retry — key contract owned by
```
api-design
```
),
```
task_type
```
/version,
```
payload
```
,
```
priority
```
,
```
scheduled_for
```
(run-not-before),
```
attempt
```
, and a
```
trace_id
```
.
Schedule spec: for recurring jobs, the cron/interval expression plus a
```
dedup_key = (job_name, scheduled_time)
```
so a double-enqueue is a no-op.
Lease contract: on dequeue a worker gets the task invisible for
```
visibility_timeout
```
; it must ack/delete on success or heartbeat to extend. Lease expiry → automatic redelivery. Failure after the retry cap → DLQ with attempt count and last error.

定时任务是一种契约。定义任务契约，而非“作业”：

任务信封：稳定的
```
task_id
```
/幂等键（重试时去重——键的定义由
```
api-design
```
负责）、
```
task_type
```
/版本、
```
payload
```
、
```
priority
```
、
```
scheduled_for
```
（最早执行时间）、
```
attempt
```
、
```
trace_id
```
。
调度规则：对于周期性作业，Cron/间隔表达式需搭配
```
dedup_key = (job_name, scheduled_time)
```
，使重复入队无效。
租赁契约：Worker出队时获得任务，并在
```
visibility_timeout
```
内该任务不可见；成功完成后需确认/删除任务，或通过心跳延长租赁时间。租赁到期→自动重发。重试次数上限后失败→转入DLQ，并携带尝试次数和最后错误信息。

Choosing a provider

选择服务商

Default to the generic recipe above. If the user names a cloud, read

references/providers/<provider>.md

for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer.

默认使用上述通用方案。若用户指定云服务商，查看

references/providers/<provider>.md

中的托管服务映射、影响决策的限制和环境特定陷阱。若该服务商无对应文档，则使用通用方案。

Diagram

示意图

To visualize the scheduler → queue → leased workers path, or the lease-expiry redelivery loop, use the in-plugin

architecture-diagram

skill. Quick inline sketch:

[scheduler] → [delay/priority queue] → workers (lease) ─expire→ requeue ─fail×N→ [DLQ]

; main path solid, the expiry/DLQ branches dashed.

若要可视化调度器→队列→租赁Worker的流程，或租赁到期重发循环，使用插件内的

architecture-diagram

技能。快速内联草图：

[scheduler] → [delay/priority queue] → workers (lease) ─expire→ requeue ─fail×N→ [DLQ]

；主路径为实线，到期/DLQ分支为虚线。

Related building blocks

References

参考资料

references/deep-dive.md
— leasing/visibility-timeout mechanics and heartbeats, distributed-cron + leader election, delay-queue implementations (sorted set, timer wheel), priority/fairness algorithms, exactly-once-effect via dedup, and Celery/Sidekiq/Airflow internals. Read when designing the scheduler in detail.
references/providers/{generic,aws,azure,gcp,temporal}.md
— service mappings, decision-changing limits, and pitfalls per environment.

references/deep-dive.md
— 租赁/可见性超时机制和心跳、分布式Cron+选主、延迟队列实现（有序集合、时间轮）、优先级/公平性算法、通过去重实现精确一次执行、Celery/Sidekiq/Airflow内部原理。详细设计调度器时阅读。
references/providers/{generic,aws,azure,gcp,temporal}.md
— 各环境下的服务映射、影响决策的限制和陷阱。