python-subprocess-and-concurrency

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Python Subprocess & Concurrency

Python Subprocess 与并发

Two topics that produce the same class of bug — a program that passes every smoke test and then hangs, leaks children, or silently drops failures under load. 29 rules in 3 categories.
A rule match is a signal, not a verdict. Concurrency advice is context-dependent: a fixed handful of tasks needs no ceremony, and a
fork
pool is still fine in a single-threaded program. Check the rule's scope line before flagging or refactoring.
这两个主题会引发同一类问题:程序通过所有 smoke test,却会在负载下卡死、泄漏子进程,或悄无声息地丢失失败结果。共 29 条规则,分为 3 个类别。
规则匹配只是一个信号,不是最终结论。并发建议取决于具体上下文:固定数量的少量任务无需任何复杂机制,而
fork
池在单线程程序中仍然可以使用。在标记或重构之前,请先检查规则的 scope 行。

Scope: what this skill owns, and what it defers

Scope:本 skill 负责什么,以及哪些内容交由其他 skill 处理

TopicOwner
subprocess
: argv, timeouts, killing trees, pipes, exit codes
this skill (
sp-
rules)
threading
,
concurrent.futures
threads
this skill (
thread-
rules)
multiprocessing
,
ProcessPoolExecutor
, start methods, IPC
this skill (
proc-
rules)
asyncio
event-loop discipline, tasks, cancellation, generators
python-async-best-practices
The offload decision itself: GIL vs threads vs process pool, default executor sizing, thread non-interruptibility
python-async-best-practices
, rule
async-no-blocking-event-loop
General Python style, typing, error handling, dataclasses
python-best-practices
Test determinism and synchronization
python-pytest
, rule
determinism-sync-not-sleep
Do not restate the deferred skills' rules here. Where they overlap, this skill states the same principle in the process/thread domain and names their rule (for example
async-bound-concurrency
for bounded fan-out,
determinism-sync-not-sleep
for concurrency tests).
Read the ownership honestly before claiming a gap.
async-no-blocking-event-loop
already covers the offload decision, the
min(32, cpu + 4)
then
process_cpu_count()
executor-sizing formula, free-threaded 3.13+ builds, and the fact that a running thread cannot be interrupted.
thread-for-io-not-cpu
,
thread-size-and-bound
and
thread-cannot-be-killed
go deeper on those for code that never touches asyncio, and each names that rule as the earlier statement rather than claiming the ground.
What is genuinely uncovered upstream, and owned here:
subprocess
in full; thread ownership, join and
excepthook
; lock discipline and pool deadlocks;
ThreadPoolExecutor
exit semantics; multiprocessing start methods, pickling, IPC and pool failure; and hang, orphan and zombie diagnosis.
主题负责方
subprocess
:argv、timeout、终止进程树、管道、退出码
本 skill(
sp-
规则)
threading
、
concurrent.futures
线程
本 skill(
thread-
规则)
multiprocessing
、
ProcessPoolExecutor
、启动方式、IPC
本 skill(
proc-
规则)
asyncio
event loop 规范、任务、取消、生成器
python-async-best-practices
卸载决策本身:GIL 与线程和进程池的选择、默认 executor 大小、线程无法中断
python-async-best-practices
,规则
async-no-blocking-event-loop
通用 Python 风格、类型标注、错误处理、dataclass
python-best-practices
测试确定性与同步
python-pytest
,规则
determinism-sync-not-sleep
不要在这里重复其他 skill 负责的规则。发生重叠时,本 skill 会说明进程和线程领域中的*相同原则,并指明相应规则(例如,有限 fan-out 对应
async-bound-concurrency
,并发测试对应
determinism-sync-not-sleep
)。
在声称存在缺口之前,应诚实判断职责边界。
async-no-blocking-event-loop
已经涵盖卸载决策、
min(32, cpu + 4)
后使用
process_cpu_count()
的 executor 大小公式、free-threaded 3.13+ 构建,以及运行中的线程无法被中断这一事实。对于完全不涉及 asyncio 的代码,
thread-for-io-not-cpu
、
thread-size-and-bound
和
thread-cannot-be-killed
会对这些问题进行更深入的说明;每条规则都将相关规则作为先前的依据,而不会重复宣称其基础。
上游确实尚未涵盖、且由本 skill 负责的内容包括:完整的
subprocess
;线程所有权、join 和
excepthook
;锁规范与池死锁;
ThreadPoolExecutor
的退出语义;multiprocessing 启动方式、pickle、IPC 和进程池故障;以及卡死、孤儿进程和僵尸进程的诊断。

Pick the model before writing code

编写代码前先选择模型

The most expensive concurrency mistake is choosing a model out of habit. Choose by what the work waits on and whether it can run in parallel.
The workModelWhy
Nothing that can overlapplain loopConcurrency is pure cost: workers and failure modes for zero gain
Many blocking I/O calls on async-capable libraries
asyncio
See
python-async-best-practices
; thousands of concurrent operations, no thread per op
Blocking I/O with libraries that have no async equivalent
ThreadPoolExecutor
Simplest tool that overlaps the wait
Pure-Python CPU work
ProcessPoolExecutor
/
multiprocessing
The GIL serializes bytecode; processes do not share it
CPU work inside a C extension that releases the GILthreadsThe GIL is not the limit for that call
CPU work, free-threaded build (
Py_GIL_DISABLED=1
)
threadsNo GIL to serialize them (PEP 703/779); measure, and confirm C extensions did not re-enable it
Thousands of small CPU tasks with little shared state
InterpreterPoolExecutor
(3.14+)
Each interpreter has its own GIL: multicore parallelism without process startup
An external program, script, or CLI
subprocess
It is a separate process whether you like it or not
Sizing facts worth knowing (verified against the 3.14 standard-library docs):
  • os.process_cpu_count()
    (3.13+) is the count usable by this process — it respects CPU affinity and container limits.
    os.cpu_count()
    is machine-wide and can return
    None
    . Use the former for pool sizing.
  • ThreadPoolExecutor(max_workers=None)
    defaults to
    min(32, (os.process_cpu_count() or 1) + 4)
    .
  • ProcessPoolExecutor(max_workers=None)
    defaults to
    os.process_cpu_count()
    , capped at 61 on Windows.
  • InterpreterPoolExecutor
    is a
    ThreadPoolExecutor
    subclass: one worker thread each, one interpreter each. Interpreters cannot share a mutable object, so it forces an explicit data contract.
代价最高的并发错误,是出于习惯选择并发模型。应根据任务在等待什么,以及任务能否并行执行来决定。
工作类型模型原因
没有任何可以重叠执行的任务普通循环并发只有成本:worker 和故障模式不会带来任何收益
在支持 async 的库上执行大量阻塞 I/O 调用
asyncio
参见
python-async-best-practices
;可并发处理数千个操作,无需每个操作占用一个线程
库没有 async 等价实现时的阻塞 I/O
ThreadPoolExecutor
重叠等待时间最简单的工具
纯 Python CPU 工作
ProcessPoolExecutor
/
multiprocessing
GIL 会串行化字节码;进程不会共享 GIL
会释放 GIL 的 C extension 内部 CPU 工作线程该调用不受 GIL 限制
CPU 工作,free-threaded 构建(
Py_GIL_DISABLED=1
)
线程没有 GIL 将其串行化(PEP 703/779);需要进行测量,并确认 C extension 没有重新启用 GIL
数千个共享状态很少的小型 CPU 任务
InterpreterPoolExecutor
(3.14+)
每个解释器都有自己的 GIL:无需进程启动开销即可实现多核并行
外部程序、脚本或 CLI
subprocess
无论是否愿意,它都是一个独立进程
值得了解的大小计算事实(已根据 3.14 标准库文档验证):
  • os.process_cpu_count()
    (3.13+)返回当前进程可用的 CPU 数量——它会考虑 CPU affinity 和容器限制。
    os.cpu_count()
    返回整台机器的 CPU 数量,并且可能返回
    None
    。应使用前者确定进程池大小。
  • ThreadPoolExecutor(max_workers=None)
    默认使用
    min(32, (os.process_cpu_count() or 1) + 4)
    。
  • ProcessPoolExecutor(max_workers=None)
    默认使用
    os.process_cpu_count()
    ,在 Windows 上上限为 61。
  • InterpreterPoolExecutor
    是
    ThreadPoolExecutor
    的子类:每个 worker 线程对应一个解释器。解释器之间无法共享可变对象,因此必须明确设计数据契约。

Quick reference

快速参考

Subprocess (
sp-
)

Subprocess(
sp-
)

  • sp-argv-not-shell
    — Pass an argv sequence; never interpolate values into
    shell=True
  • sp-check-and-capture
    —
    check=True
    (or read
    returncode
    explicitly);
    check
    defaults to
    False
  • sp-timeout-always
    — Every wait has a timeout;
    run(timeout=)
    kills only the direct child
  • sp-kill-the-process-tree
    —
    start_new_session=True
    +
    os.killpg
    to actually stop descendants
  • sp-communicate-not-wait
    — Use
    communicate()
    with pipes;
    .wait()
    /
    .read()
    deadlocks, and a surviving descendant keeps EOF from arriving
  • sp-popen-only-when-needed
    —
    run()
    by default;
    with Popen
    waits, it does not kill
  • sp-close-stdin
    —
    stdin=DEVNULL
    for non-interactive children
  • sp-explicit-env-and-cwd
    — Full executable path, explicit
    cwd
    and
    env
  • sp-explicit-text-encoding
    —
    encoding="utf-8"
    , never the locale by accident
  • sp-no-preexec-fn
    —
    preexec_fn
    is thread-unsafe; use
    start_new_session
    /
    process_group
  • sp-bound-child-fanout
    — One child per input item is unbounded fan-out
  • sp-drop-legacy-apis
    — Not
    os.system
    ,
    os.popen
    ,
    os.spawn*
    ,
    getstatusoutput
  • sp-async-children
    — In async code use
    create_subprocess_exec
    +
    asyncio.timeout
  • sp-argv-not-shell
    — 传入 argv 序列;绝不要将值插入
    shell=True
  • sp-check-and-capture
    — 使用
    check=True
    (或显式读取
    returncode
    );
    check
    默认为
    False
  • sp-timeout-always
    — 每次等待都必须设置 timeout;
    run(timeout=)
    只会终止直接子进程
  • sp-kill-the-process-tree
    — 使用
    start_new_session=True
    +
    os.killpg
    ,才能真正停止所有后代进程
  • sp-communicate-not-wait
    — 带管道时使用
    communicate()
    ;
    .wait()
    /
    .read()
    会导致死锁,而残留的后代进程会阻止 EOF 到达
  • sp-popen-only-when-needed
    — 默认使用
    run()
    ;使用
    with Popen
    时只会等待,不会终止进程
  • sp-close-stdin
    — 对非交互式子进程使用
    stdin=DEVNULL
  • sp-explicit-env-and-cwd
    — 使用完整的 executable 路径,并显式指定
    cwd
    和
    env
  • sp-explicit-text-encoding
    — 使用
    encoding="utf-8"
    ,不要意外采用 locale 编码
  • sp-no-preexec-fn
    —
    preexec_fn
    不具备线程安全性;应使用
    start_new_session
    /
    process_group
  • sp-bound-child-fanout
    — 为每个输入项创建一个子进程会造成无限 fan-out
  • sp-drop-legacy-apis
    — 不使用
    os.system
    、
    os.popen
    、
    os.spawn*
    、
    getstatusoutput
  • sp-async-children
    — 在 async 代码中使用
    create_subprocess_exec
    +
    asyncio.timeout

Threads (
thread-
)

线程(
thread-
)

  • thread-cannot-be-killed
    — No stop/suspend; stop cooperatively, and not with daemon threads
  • thread-own-and-join
    — Join every thread or use an executor; a failed thread prints and vanishes
  • thread-size-and-bound
    — Set
    max_workers
    ; bound submission the way
    async-bound-concurrency
    bounds fan-out
  • thread-never-wait-on-a-future
    — A worker waiting on another future deadlocks the pool
  • thread-for-io-not-cpu
    — Threads do not speed up pure-Python CPU work
  • thread-share-by-message
    — Pass data, guard shared mutable state, keep locks short and ordered
  • thread-no-pool-for-long-running-work
    — The exit handler joins threads before
    atexit
    ; a pool is not a scheduler
  • thread-explicit-shutdown
    — Context manager or explicit
    shutdown()
    ;
    cancel_futures
    only reaches queued work
  • thread-cannot-be-killed
    — 无法停止或挂起线程;应采用协作式停止,而且不要使用 daemon thread
  • thread-own-and-join
    — Join 每个线程,或者使用 executor;线程失败后会打印错误并消失
  • thread-size-and-bound
    — 设置
    max_workers
    ;像
    async-bound-concurrency
    限制 fan-out 一样限制任务提交量
  • thread-never-wait-on-a-future
    — worker 等待另一个 future 会导致进程池死锁
  • thread-for-io-not-cpu
    — 线程无法加速纯 Python CPU 工作
  • thread-share-by-message
    — 传递数据,保护共享可变状态,锁应保持简短且获取顺序固定
  • thread-no-pool-for-long-running-work
    — 退出处理会在
    atexit
    前 join 线程;进程池不是调度器
  • thread-explicit-shutdown
    — 使用 context manager 或显式调用
    shutdown()
    ;
    cancel_futures
    只能取消排队中的工作

Processes (
proc-
)

进程(
proc-
)

  • proc-choose-start-method
    — Name the start method;
    fork
    is no longer a default anywhere
  • proc-main-guard
    —
    if __name__ == "__main__":
    plus
    freeze_support()
  • proc-top-level-and-picklable
    — Workers are module-level functions; args and results are pickled
  • proc-inherit-dont-ship
    — Set up each worker in
    initializer
    ; inherit resources instead of shipping them
  • proc-message-pass
    — Queues for data,
    Pipe
    for one-to-one, proxies sparingly
  • proc-expect-a-broken-pool
    — A killed worker breaks the whole pool; catch
    BrokenExecutor
  • proc-stop-cooperatively
    — Terminating a worker corrupts shared queues; signals are main-thread-only
  • proc-bound-and-chunk
    — Size the pool yourself;
    chunksize
    matters,
    map
    collects eagerly
  • proc-choose-start-method
    — 明确指定启动方式;
    fork
    已不再是任何平台的默认方式
  • proc-main-guard
    — 使用
    if __name__ == "__main__":
    ,并调用
    freeze_support()
  • proc-top-level-and-picklable
    — Worker 必须是模块级函数;参数和结果会被 pickle
  • proc-inherit-dont-ship
    — 在
    initializer
    中设置每个 worker;应继承资源,而不是传输资源
  • proc-message-pass
    — 使用 Queue 传递数据,使用
    Pipe
    进行一对一通信,谨慎使用 proxy
  • proc-expect-a-broken-pool
    — worker 被终止会破坏整个进程池;应捕获
    BrokenExecutor
  • proc-stop-cooperatively
    — 强制终止 worker 会破坏共享 Queue;信号只能在主线程中处理
  • proc-bound-and-chunk
    — 自行确定进程池大小;
    chunksize
    很重要,
    map
    会立即收集结果

The patterns that carry most of the weight

最关键的模式

Every external command, in one shape. Argv, closed stdin, explicit encoding, timeout, checked exit code:
python
import subprocess

def git_rev(path: str) -> str:
    result = subprocess.run(
        ["/usr/bin/git", "-C", path, "rev-parse", "HEAD"],
        stdin=subprocess.DEVNULL,
        capture_output=True,
        text=True,
        encoding="utf-8",
        errors="replace",
        timeout=30,
        check=True,
    )
    return result.stdout.strip()
A timeout that stops the whole tree.
run(timeout=)
kills only the direct child; anything it spawned keeps running and keeps holding your resources.
python
import os
import signal
import subprocess

def run_tree(argv: list[str], *, timeout: float, grace: float = 5.0) -> int:
    # start_new_session=True puts the child in its own session, so its process
    # group id equals its pid: one killpg stops the child and every descendant.
    proc = subprocess.Popen(argv, start_new_session=True, stdin=subprocess.DEVNULL)
    try:
        return proc.wait(timeout=timeout)
    except subprocess.TimeoutExpired:
        os.killpg(proc.pid, signal.SIGTERM)      # ask the whole group to stop
        try:
            proc.wait(timeout=grace)             # give it a moment to exit cleanly
        except subprocess.TimeoutExpired:
            os.killpg(proc.pid, signal.SIGKILL)  # then stop asking
            proc.wait()                          # always reap
        raise
Bounded child fan-out. One child per input item is a fork bomb dressed as a loop:
python
from concurrent.futures import ThreadPoolExecutor
import subprocess

def run_all(argv_list: list[list[str]], *, max_parallel: int = 8) -> list[str]:
    def one(argv: list[str]) -> str:
        done = subprocess.run(
            argv, capture_output=True, text=True, encoding="utf-8",
            timeout=60, check=True,
        )
        return done.stdout

    # max_workers bounds how many children exist at once. Same principle as
    # async-bound-concurrency, applied to processes instead of coroutines.
    with ThreadPoolExecutor(max_workers=max_parallel) as executor:
        return list(executor.map(one, argv_list))
Shutdown you can predict. A daemon thread is killed mid-work at interpreter exit; a worker pool joins what it accepted.
python
from concurrent.futures import ThreadPoolExecutor

def handle(item: str) -> None: ...

with ThreadPoolExecutor(max_workers=4) as executor:   # shutdown(wait=True) on exit
    futures = [executor.submit(handle, item) for item in items]
    for future in futures:
        future.result()   # never let a worker's exception disappear
Per-worker setup, results as plain data. Both executors give you a one-time per-worker hook; use it instead of reopening a connection per task, and return data instead of shipping live objects:
python
import multiprocessing
from concurrent.futures import ProcessPoolExecutor

_conn = None                      # assigned once per worker by init_worker()

def init_worker(dsn: str) -> None:
    global _conn
    _conn = connect(dsn)          # opened once per worker process, not once per task

def score(row_id: int) -> float:
    return _conn.query(row_id)    # must be importable at module top level to be picklable

if __name__ == "__main__":        # required by the spawn and forkserver start methods
    ctx = multiprocessing.get_context("spawn")   # name it; never inherit the default blindly
    with ProcessPoolExecutor(max_workers=8, mp_context=ctx,
                             initializer=init_worker, initargs=(dsn,)) as executor:
        results = list(executor.map(score, row_ids, chunksize=200))
所有外部命令都采用同一种形式。 使用 argv、关闭 stdin、显式指定 encoding、设置 timeout,并检查退出码:
python
import subprocess

def git_rev(path: str) -> str:
    result = subprocess.run(
        ["/usr/bin/git", "-C", path, "rev-parse", "HEAD"],
        stdin=subprocess.DEVNULL,
        capture_output=True,
        text=True,
        encoding="utf-8",
        errors="replace",
        timeout=30,
        check=True,
    )
    return result.stdout.strip()
能够终止整个进程树的 timeout。
run(timeout=)
只会终止直接子进程;子进程创建的任何程序仍会继续运行,并持续占用资源。
python
import os
import signal
import subprocess

def run_tree(argv: list[str], *, timeout: float, grace: float = 5.0) -> int:
    # start_new_session=True puts the child in its own session, so its process
    # group id equals its pid: one killpg stops the child and every descendant.
    proc = subprocess.Popen(argv, start_new_session=True, stdin=subprocess.DEVNULL)
    try:
        return proc.wait(timeout=timeout)
    except subprocess.TimeoutExpired:
        os.killpg(proc.pid, signal.SIGTERM)      # ask the whole group to stop
        try:
            proc.wait(timeout=grace)             # give it a moment to exit cleanly
        except subprocess.TimeoutExpired:
            os.killpg(proc.pid, signal.SIGKILL)  # then stop asking
            proc.wait()                          # always reap
        raise
限制子进程 fan-out。 为每个输入项创建一个子进程,就是伪装成循环的 fork bomb:
python
from concurrent.futures import ThreadPoolExecutor
import subprocess

def run_all(argv_list: list[list[str]], *, max_parallel: int = 8) -> list[str]:
    def one(argv: list[str]) -> str:
        done = subprocess.run(
            argv, capture_output=True, text=True, encoding="utf-8",
            timeout=60, check=True,
        )
        return done.stdout

    # max_workers bounds how many children exist at once. Same principle as
    # async-bound-concurrency, applied to processes instead of coroutines.
    with ThreadPoolExecutor(max_workers=max_parallel) as executor:
        return list(executor.map(one, argv_list))
可预测的关闭流程。 daemon thread 会在解释器退出时被中途终止;worker pool 则会等待已接受的任务完成。
python
from concurrent.futures import ThreadPoolExecutor

def handle(item: str) -> None: ...

with ThreadPoolExecutor(max_workers=4) as executor:   # shutdown(wait=True) on exit
    futures = [executor.submit(handle, item) for item in items]
    for future in futures:
        future.result()   # never let a worker's exception disappear
每个 worker 独立初始化,结果使用纯数据。 两种 executor 都提供一次性 per-worker hook;应使用它,而不是为每个任务重新建立连接,并且应返回数据,而不是传输活对象:
python
import multiprocessing
from concurrent.futures import ProcessPoolExecutor

_conn = None                      # assigned once per worker by init_worker()

def init_worker(dsn: str) -> None:
    global _conn
    _conn = connect(dsn)          # opened once per worker process, not once per task

def score(row_id: int) -> float:
    return _conn.query(row_id)    # must be importable at module top level to be picklable

if __name__ == "__main__":        # required by the spawn and forkserver start methods
    ctx = multiprocessing.get_context("spawn")   # name it; never inherit the default blindly
    with ProcessPoolExecutor(max_workers=8, mp_context=ctx,
                             initializer=init_worker, initargs=(dsn,)) as executor:
        results = list(executor.map(score, row_ids, chunksize=200))

Debugging a hang

调试卡死问题

A hang is a missing timeout, a full pipe, or a lock. In that order.
  • Get a stack out of a live process:
    faulthandler.dump_traceback_later(60, exit=True)
    , or run with
    python -X faulthandler
    . It is implemented in C, so it still reports during a deadlock in Python code.
  • py-spy dump --pid <pid>
    prints every thread's stack without restarting the process;
    py-spy top --pid <pid>
    shows what is burning CPU;
    --locals
    adds frame locals. In containers it needs the
    SYS_PTRACE
    capability.
  • threading.enumerate()
    lists thread objects that are still alive;
    multiprocessing.active_children()
    does the same for children.
  • Orphans and zombies point at the opposite mistakes: a child outliving its parent means the wrong thing was killed (
    sp-kill-the-process-tree
    ), and an unreaped child means nobody joined it (
    thread-own-and-join
    ,
    proc-main-guard
    ).
Commands and diagnosis order:
references/debugging-hangs.md
.
卡死通常源于缺少 timeout、管道已满或锁。应按此顺序排查。
  • 从运行中的进程获取 stack:使用
    faulthandler.dump_traceback_later(60, exit=True)
    ,或者通过
    python -X faulthandler
    运行。它以 C 实现,因此即使 Python 代码发生死锁,仍能输出报告。
  • py-spy dump --pid <pid>
    可以在不重启进程的情况下打印每个线程的 stack;
    py-spy top --pid <pid>
    用于显示 CPU 占用热点;
    --locals
    会增加 frame locals。在容器中运行时,它需要
    SYS_PTRACE
    capability。
  • threading.enumerate()
    会列出仍然存活的线程对象;
    multiprocessing.active_children()
    则用于列出仍然存活的子进程。
  • 孤儿进程和僵尸进程指向相反的错误:子进程比父进程活得更久,说明终止了错误的进程(
    sp-kill-the-process-tree
    );子进程未被回收则说明没有人 join 它(
    thread-own-and-join
    、
    proc-main-guard
    )。
命令与诊断顺序参见
references/debugging-hangs.md
。

Review checklist

审查清单

  • Every
    subprocess
    call passes an argv sequence, and
    shell=True
    appears nowhere.
  • Every call sets
    check=True
    or inspects
    returncode
    deliberately.
  • Every call has a timeout, and a timed-out child cannot leave descendants running or holding its pipes open.
  • No
    .read()
    ,
    .stdin.write()
    , or
    .wait()
    against a
    PIPE
    ;
    communicate()
    is used.
  • encoding
    is explicit when text mode is used.
  • stdin=DEVNULL
    (or explicit
    input=
    ) for anything non-interactive.
  • Executable path,
    cwd
    , and
    env
    are explicit where the result must be reproducible.
  • No
    preexec_fn
    .
  • Fan-out is bounded: at most N children or N workers, chosen from a real downstream limit.
  • No
    os.system
    ,
    os.popen
    ,
    os.spawn*
    , or
    subprocess.getstatusoutput
    .
  • Every thread is joined, or lives in an executor that is shut down.
  • No daemon thread owns work whose interruption would corrupt state.
  • No worker waits on another task's future.
  • Shared mutable state across threads is message-passed or guarded; locks are acquired in one global order and never held across a blocking call.
  • CPU-bound work in threads is justified by a free-threaded build or a GIL-releasing extension.
  • Pool sizing is explicit, and uses
    os.process_cpu_count()
    (3.13+) rather than
    os.cpu_count()
    .
  • The multiprocessing start method is named explicitly, and
    fork
    is not used in a threaded process.
  • Worker functions are module-level and picklable; args and results are plain data.
  • if __name__ == "__main__":
    guards the entry point of anything that spawns processes.
  • BrokenExecutor
    /
    BrokenProcessPool
    is handled where a worker can die.
  • Shutdown is cooperative for threads and processes;
    terminate()
    is not used on workers holding shared queues.
  • 每个
    subprocess
    调用都传入 argv 序列,并且任何地方都没有
    shell=True
    。
  • 每个调用都设置
    check=True
    ,或有意检查
    returncode
    。
  • 每个调用都设置 timeout,且超时后子进程无法留下仍在运行或占用管道的后代进程。
  • 不对
    PIPE
    使用
    .read()
    、
    .stdin.write()
    或
    .wait()
    ;改用
    communicate()
    。
  • 使用文本模式时显式指定
    encoding
    。
  • 对任何非交互式进程使用
    stdin=DEVNULL
    (或显式提供
    input=
    )。
  • 当结果必须可复现时,显式指定 executable 路径、
    cwd
    和
    env
    。
  • 不使用
    preexec_fn
    。
  • Fan-out 受到限制:最多存在 N 个子进程或 N 个 worker,数量应根据真实的下游限制确定。
  • 不使用
    os.system
    、
    os.popen
    、
    os.spawn*
    或
    subprocess.getstatusoutput
    。
  • 每个线程都被 join,或位于已关闭的 executor 中。
  • 没有 daemon thread 承担一旦被中断就会破坏状态的工作。
  • 没有 worker 等待另一个任务的 future。
  • 线程之间的共享可变状态通过消息传递或受锁保护;所有位置都按同一个全局顺序获取锁,且绝不跨阻塞调用持有锁。
  • 线程中的 CPU 密集型工作只有在 free-threaded 构建或会释放 GIL 的 extension 中才有合理性。
  • 明确指定进程池大小,并使用
    os.process_cpu_count()
    (3.13+),而不是
    os.cpu_count()
    。
  • 明确指定 multiprocessing 启动方式,并且不在多线程进程中使用
    fork
    。
  • Worker 函数位于模块级且可被 pickle;参数和结果均为纯数据。
  • 任何创建进程的入口点都由
    if __name__ == "__main__":
    保护。
  • 在 worker 可能死亡的位置处理
    BrokenExecutor
    /
    BrokenProcessPool
    。
  • 线程和进程采用协作式关闭;绝不对持有共享 Queue 的 worker 使用
    terminate()
    。

References

参考资料

  • references/subprocess.md
    — the
    sp-
    rules with failure modes, code, and platform notes.
  • references/threads.md
    — the
    thread-
    rules: GIL, free-threading, ownership, locks, pool behavior.
  • references/processes.md
    — the
    proc-
    rules: start methods, pickling, IPC, pool lifecycle.
  • references/debugging-hangs.md
    — hang, deadlock, orphan, and zombie diagnosis with commands.
Run
scripts/verify-recipes.py
to re-check this skill's recipes and version claims on the interpreter you have (repo tooling, wired into
npm test
; execute it, do not read it into context). It prints one line per check naming the rule it backs, skips POSIX-only checks off POSIX, and asserts the documented API floor for your Python version.
Standards this follows: PEP 324 (subprocess), PEP 703 and PEP 779 (free-threading; officially supported in 3.14), PEP 3148 (
concurrent.futures
), PEP 734 (interpreters). Cross-skill:
python-async-best-practices
,
python-best-practices
,
python-pytest
.
  • references/subprocess.md
    —
    sp-
    规则,包括故障模式、代码和平台注意事项。
  • references/threads.md
    —
    thread-
    规则:GIL、free-threading、所有权、锁和进程池行为。
  • references/processes.md
    —
    proc-
    规则:启动方式、pickle、IPC 和进程池生命周期。
  • references/debugging-hangs.md
    — 使用命令诊断卡死、死锁、孤儿进程和僵尸进程。
运行
scripts/verify-recipes.py
,在当前解释器上重新检查本 skill 的示例和版本声明(repo tooling,已接入
npm test
;直接执行,不要将其读入上下文)。它会为每项检查输出一行并注明对应规则,在非 POSIX 系统上跳过 POSIX 专属检查,并断言当前 Python 版本所对应的文档 API 最低版本。
遵循的标准:PEP 324(subprocess)、PEP 703 和 PEP 779(free-threading;3.14 起正式支持)、PEP 3148(
concurrent.futures
)、PEP 734(解释器)。跨 skill:
python-async-best-practices
、
python-best-practices
、
python-pytest
。