Back to Details

runpod-serverless-deploy

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Serverless deploy

Serverless 部署

Turn any deployable source into a queue-backed Serverless endpoint proven with one real job; the user leaves knowing what exists and what it bills. Steps: read the source, decide engine and hardware, provision, warm, smoke-test one real job, hand back the URL and a 
curl
, state what is billing. Tear down only when the user framed the deploy as a throwaway test or asks for cleanup.
Two decisions are yours, unprompted: the engine family (the right worker image for this modality and model format) and the provisioning path (bake weights in, download-on-start, or a network volume; each costs a different cold start). Size hardware from the model's real metadata, never a guess. Among suitable workers prefer the smallest, most-proven image for a smoke-able deploy (a giant image makes the first job a many-minute cold start; the smoke test proves the journey, not maximal quality), and name the trade-off if the request implies a heavier engine. Creating an endpoint bills: state the GPU class's hourly cost before creating; the smoke job costs money too.
For code-first Python (writing endpoint functions in a project and iterating with the Runpod SDK's 
rp flash dev
loop), this MCP-only bundle has no path. Say so instead of improvising one. This skill deploys an existing source (Hub repo, container image, ComfyUI workflow, Hub release) to an endpoint through the MCP tools.
将任何可部署来源变成由队列驱动的 Serverless endpoint，并通过一个真实 job 加以验证；用户最终会清楚现有资源和计费情况。步骤：读取来源、决定 engine 和硬件、完成置备、预热、对一个真实 job 进行 smoke test，然后交回 URL 和一条 
curl
，并说明计费内容。仅当用户明确表示这是一次 throwaway test 或要求清理时，才进行拆除。
以下两项决定由你主动作出，无需先询问用户：engine 家族（适合该 modality 和 model format 的 worker 镜像）以及置备路径（将权重烘焙进镜像、启动时下载或使用 network volume；三者的 cold start 成本不同）。必须依据模型的真实 metadata 确定硬件规格，绝不能猜测。在合适的 worker 中，优先选择体积最小且经过充分验证、适合 smoke test 的镜像（巨型镜像会让首个 job 的 cold start 持续数分钟；smoke test 验证的是整个流程，而不是最高质量）；如果用户的需求意味着必须使用更重的 engine，应明确说明这一取舍。创建 endpoint 会产生费用：创建前必须说明所选 GPU class 的小时费率；smoke job 同样会产生费用。
对于 code-first Python（在项目中编写 endpoint 函数，并使用 Runpod SDK 的 
rp flash dev
loop 迭代），该 MCP-only bundle 没有对应路径。应直接说明，而不要临时拼凑一个方案。本 skill 通过 MCP tools 将现有来源（Hub 仓库、容器镜像、ComfyUI workflow、Hub release）部署为 endpoint。

Tool binding

Tool 绑定

Read the catalog before creating; create only what the user asked to deploy. A tool named here missing from the session: say so and use the nearest read; never invent a result.
CapabilityTool
GPU catalog, price, stock (the sizing math is 
runpod-discovery
's); per-DC stock (
dataCenters[]
); CUDA versions with capacity (
cudaVersions
)
list-gpu-types
/ 
get-gpu-type
(
include:["AVAILABILITY"]
, 
product:["SERVERLESS"]
); 
list-data-centers
(
include:["GPU_AVAILABILITY"]
), 
get-data-center
; 
get-capacity
for the CUDA matrix
Hub / community catalog with the release config (hardware, env schema); deploy a release in one call; pin a release / read history
list-hub-repos
(
includeConfig:true
); 
deploy-hub-repo
; 
list-endpoint-releases
Template (image, disk, ports, env, registry, mounts); endpoint (GPU pools or CPU flavors, worker min/max, scaler, data centers, network volumes, 
timeout
); network volume for large weights in a GPU-co-located DC
create-template
; 
create-endpoint
; 
create-network-volume
Multi-GPU / pin to one card (
gpu.count
, pool plus 
excludedTypes
)
set-endpoint-gpus
(
pools
, 
excludeGpuTypeIds
, 
gpuCount
; builds the exclusion list, preserves other settings), or 
update-endpoint
with 
gpu.pools
and 
gpu.excludedTypes
together
Health / warmup; per-worker detail (a crash-looping container can still report RUNNING or INITIALIZING)
endpoint-health
; 
list-endpoint-workers
Smoke test, blocking to terminal
run-endpoint
, then 
get-job-status
(
wait:300000
); 
runsync-endpoint
only for a warm endpoint and a fast job (its result is kept for one minute)
Worker logs (model-load / crash evidence)
stream-worker-logs
(
source=system
for the container lifecycle). Without it, 
endpoint-health
+ 
list-endpoint-workers
narrow the cause but cannot settle a crash loop
Private-image registry credential; inspect; teardown
create-registry
, 
get-registry
; 
get-endpoint
, 
get-template
; 
delete-endpoint
, 
delete-template
, 
delete-network-volume
创建前先读取目录；只创建用户要求部署的内容。如果当前会话中缺少此处指定的工具，应明确说明并使用最接近的只读能力；绝不虚构结果。
能力Tool
GPU 目录、价格、库存（规格测算由 
runpod-discovery
负责）；各 DC 库存（
dataCenters[]
）；具有容量的 CUDA 版本（
cudaVersions
）
list-gpu-types
/ 
get-gpu-type
（
include:["AVAILABILITY"]
、
product:["SERVERLESS"]
）；
list-data-centers
（
include:["GPU_AVAILABILITY"]
）、
get-data-center
；
get-capacity
用于查询 CUDA 矩阵
Hub／社区目录及 release 配置（硬件、env schema）；一次调用部署 release；固定 release／读取历史记录
list-hub-repos
（
includeConfig:true
）；
deploy-hub-repo
；
list-endpoint-releases
Template（镜像、磁盘、端口、env、registry、挂载）；endpoint（GPU pools 或 CPU flavors、worker min/max、scaler、数据中心、network volumes、
timeout
）；在与 GPU 同址的 DC 中为大型权重创建 network volume
create-template
；
create-endpoint
；
create-network-volume
Multi-GPU／固定到单张卡（
gpu.count
、pool 加 
excludedTypes
）
set-endpoint-gpus
（
pools
、
excludeGpuTypeIds
、
gpuCount
；构建排除列表并保留其他设置），或通过 
update-endpoint
同时设置 
gpu.pools
和 
gpu.excludedTypes
健康状态／预热；worker 详情（不断崩溃重启的容器仍可能报告 RUNNING 或 INITIALIZING）
endpoint-health
；
list-endpoint-workers
Smoke test，阻塞至终态
run-endpoint
，然后执行 
get-job-status
（
wait:300000
）；
runsync-endpoint
仅用于已预热的 endpoint 和快速 job（结果只保留一分钟）
Worker 日志（模型加载／崩溃证据）
stream-worker-logs
（容器生命周期使用 
source=system
）。没有该工具时，
endpoint-health
+ 
list-endpoint-workers
可以缩小原因范围，但无法最终确认崩溃循环
私有镜像 registry 凭证；检查；拆除
create-registry
、
get-registry
；
get-endpoint
、
get-template
；
delete-endpoint
、
delete-template
、
delete-network-volume

Phase 0 — identify the source and pick the engine (unprompted decision)

Phase 0 — 识别来源并选择 engine（主动决策）

Read and classify the source you were given: the engine follows the modality and model format, not a default. Never serve a non-text-generation task on a vLLM or ComfyUI worker; transcription, TTS, OCR and embeddings are not chat-worker jobs.
Source / modalityEngine family → worker image
Text-generation LLM, 
safetensors
weights		vLLM or SGLang → 
runpod/worker-v1-vllm
(or an SGLang worker)
Text-generation LLM, 
GGUF
quantized weights		llama.cpp / Ollama → a GGUF-capable worker, not vLLM
Image/diffusion 
workflow.json
ComfyUI → 
runpod/worker-comfyui
(reference/comfyui-serverless.md)
Speech-to-text (transcription)a Whisper-family model → a speech-to-text worker, not an LLM worker
Text-to-speech / voice cloninga TTS model → a TTS worker
Speaker diarizationa diarization pipeline → a diarization worker from the Hub or public catalog
OCR / document extractiona classic OCR engine → an OCR worker, not a VLM chat worker
Text embeddingsan embeddings server (e.g. text-embeddings-inference) → an embeddings worker
A custom handler you were givenyour own image → a custom container image
  • Format detection. Inspect the repo's files first: 
    *.safetensors
    / 
    *.bin
    → vLLM/SGLang; 
    *.gguf
    → llama.cpp/Ollama. GGUF on worker-vllm is a common wrong-engine mistake: vLLM will not load it. State the detected format and the engine it implies in the final message.
  • OCR sizing. For an OCR smoke deploy, default to a classic lightweight OCR worker (16–24 GB). A VLM-based OCR (7B+ params) is a big-model deploy: size it like one (48 GB class) or don't pick it for a smoke test.
  • Community-template source. When the modality has no first-party Runpod worker, browse the Hub / public catalog for a community (non-Runpod) template. Name the repo and its owner from the listing, never from memory. Once one fits, deploy it and run one real job: don't stop at "I found X", and don't block on a credential the config makes optional (phase 2's runtime-optional-token rule); request the capability through the job input the config documents. Only if nothing fits, say "none found" and stop.
  • Custom image. The user's image is the engine: deploy exactly the image named, never a substitute. Missing required env → phase 2's model-default rule, not a question back; the observed worker states are the report. After the deploy read the worker states (
    list-endpoint-workers
    or 
    endpoint-health
    ): for a custom image the workers' health IS the deliverable, and one completed job hides a crash-looping sibling, so report a crash-looping custom image as failing. A private registry needs a registry credential (create/reference it; see 
    runpod-lifecycle-crud
    ); never echo the registry password. For a load-balancer (custom-HTTP) endpoint, read reference/scenarios.md (guide id 
    runpod-serverless-deploy/scenarios
    ) before the create: the worker must pass the HTTP health check or the endpoint sits at zero ready workers, and the type cannot be changed later.
读取并分类用户提供的来源：engine 应由 modality 和 model format 决定，不能套用默认值。绝不能用 vLLM 或 ComfyUI worker 提供非文本生成任务；转录、TTS、OCR 和 embeddings 都不是 chat worker 的任务。
来源／modalityEngine 家族 → worker 镜像
文本生成 LLM，
safetensors
权重		vLLM 或 SGLang → 
runpod/worker-v1-vllm
（或 SGLang worker）
文本生成 LLM，
GGUF
量化权重		llama.cpp／Ollama → 支持 GGUF 的 worker，不能使用 vLLM
图像／diffusion 
workflow.json
ComfyUI → 
runpod/worker-comfyui
（reference/comfyui-serverless.md）
Speech-to-text（转录）Whisper 系列模型 → speech-to-text worker，而不是 LLM worker
Text-to-speech／voice cloningTTS 模型 → TTS worker
Speaker diarizationdiarization pipeline → 来自 Hub 或公共目录的 diarization worker
OCR／文档提取传统 OCR engine → OCR worker，而不是 VLM chat worker
文本 embeddingsembeddings server（例如 text-embeddings-inference）→ embeddings worker
用户提供的自定义 handler用户自己的镜像 → 自定义容器镜像
  • 格式检测。 首先检查仓库文件：
    *.safetensors
    / 
    *.bin
    → vLLM／SGLang；
    *.gguf
    → llama.cpp／Ollama。将 GGUF 部署到 worker-vllm 是常见的 engine 选择错误：vLLM 无法加载它。最终消息中必须说明检测到的格式及其对应的 engine。
  • OCR 规格。 对 OCR smoke deploy，默认使用轻量级传统 OCR worker（16–24 GB）。基于 VLM 的 OCR（7B 以上参数）属于大模型部署：应按大模型规格配置（48 GB class），否则不要将其用于 smoke test。
  • 社区模板来源。 如果某种 modality 没有 Runpod 第一方 worker，应浏览 Hub／公共目录，寻找社区（非 Runpod）模板。仓库名称及其 owner 必须来自列表信息，绝不能凭记忆编造。找到合适模板后，必须部署并运行一个真实 job：不能停在“我找到了 X”，也不能因为配置将某项凭证设为可选而阻塞部署（参见 Phase 2 的 runtime-optional-token 规则）；应按照配置文档，通过 job 输入请求该能力。只有在没有任何匹配项时，才回答“none found”并停止。
  • 自定义镜像。 用户提供的镜像就是 engine：准确部署指定的镜像，绝不替换。缺少必需 env 时，遵循 Phase 2 的 model-default 规则，而不是反问用户；观察到的 worker 状态就是报告依据。部署后读取 worker 状态（
    list-endpoint-workers
    或 
    endpoint-health
    ）：对于自定义镜像，worker 的健康状态就是交付物；一个已完成 job 会掩盖另一个不断崩溃的 sibling，因此必须将发生崩溃循环的自定义镜像报告为失败。私有 registry 需要 registry 凭证（创建或引用它；参见 
    runpod-lifecycle-crud
    ）；绝不回显 registry 密码。对于负载均衡器（custom-HTTP） endpoint，创建前必须阅读 reference/scenarios.md（guide id 
    runpod-serverless-deploy/scenarios
    ）：worker 必须通过 HTTP health check，否则 endpoint 会一直没有 ready worker，而且该类型之后无法更改。

Phase 1 — size the hardware (read the catalog, cite 
runpod-discovery
)

Phase 1 — 确定硬件规格（读取目录，引用 
runpod-discovery
）

runpod-discovery
owns the params×precision + KV/overhead VRAM math; apply its result to a real GPU pick.
  • Read the model's real metadata (parameter count, precision; for diffusion the checkpoint/LoRA/VAE sizes), not the repo name alone when the config is readable. Big-model sizing is shown arithmetic, never a bare total: the parameter count and where you read it, the bytes per parameter at the chosen precision, the weights total, the KV-cache/overhead allowance, and the GPU decision they add up to.
  • Pick the smallest GPU class whose VRAM fits, then confirm live stock: 
    include:["AVAILABILITY"]
    goes with 
    product:["SERVERLESS"]
    (stock differs by product), and the same read gives per-data-center stock and, in 
    cudaVersions
    , which CUDA versions have capacity. Pin the CUDA floor on the create: 
    gpu.minCudaVersion
    at the image's CUDA line (12.8 for a CUDA 12 image; 13.0 only for an image built on CUDA 13), or 
    gpu.allowedCudaVersions
    for an exact set. They are mutually exclusive (400 if both are sent), and the floor must have stock in 
    cudaVersions
    . Never create into a GPU class with no stock: a scale-to-zero endpoint that can never acquire a worker looks deployed but never runs.
  • Pinning to one card (the user wants an exact GPU): select the pool that contains the card and list the pool's other GPU types in 
    gpu.excludedTypes
    (
    gpu.pools
    takes pool ids, not GPU type ids; at least one type must remain), on the create or with 
    set-endpoint-gpus
    afterwards. The read-back quotes pools and exclusions, and every other setting is confirmed unchanged from the read.
  • Multi-GPU when one card can't hold the model (required VRAM > the largest single card with stock): GPUs-per-worker > 1 plus tensor parallelism in the worker env (e.g. vLLM's tensor-parallel setting = GPU count). Say you are going multi-GPU and why, that it costs proportionally more per hour, and quote the largest in-stock single card (VRAM, live stock, $/hr) against the multi-GPU pick before committing.
  • Pin the pool on Hub deploys. 
    deploy-hub-repo
    inherits the release's GPU pool list when 
    gpuIds
    is omitted, which can silently acquire a top-tier card for a task a 24 GB card serves, at several times the hourly rate. Pass 
    gpuIds
    from YOUR sizing on every Hub deploy unless the release config pins exactly the pool you chose.
  • CPU serverless. 
    create-endpoint
    takes one hardware choice: the 
    gpu
    object (pools and count) or the 
    cpu
    array of CPU flavors sized from 
    list-cpu-types
    . For a CPU workload send 
    cpu
    and no 
    gpu
    (never force a GPU pool onto it), and quote the CPU flavor's hourly price.
  • Set the values you mean. A REST create defaults to FlashBoot 
    OFF
    , 
    workers.max
    3 and 
    workers.idleTimeout
    10 s: set 
    flashboot
    and the worker bounds explicitly. Always send 
    timeout
     (per-request execution limit, in ms) sized to the slowest real job: a v2 create without it comes back 
    "timeout": 0
    (live-verified 2026-10-29; why it matters). On a Hub deploy, read 
    timeout
    back and set it with 
    update-endpoint
    if it is 0. The sum of 
    workers.max
    across the account is capped by a balance-tiered quota (5 to start), checked at create, so a second endpoint at the default of 3 can be refused.
  • State the cost. Quote the chosen GPU's hourly rate (×GPU count) from the catalog read before creating. Creating the endpoint and running the smoke test both bill.
runpod-discovery
负责参数量×精度以及 KV／额外开销的 VRAM 测算；将其结果用于选择真实的 GPU。
  • 读取模型的真实 metadata（参数量、精度；对于 diffusion 则包括 checkpoint／LoRA／VAE 大小），不能只根据仓库名称判断，即使配置可读也必须读取。**大模型规格必须展示计算过程，不能只给出总数：**说明参数量及其读取位置、所选精度下每个参数占用的字节数、权重总量、KV-cache／额外开销预留，以及由这些数据得出的 GPU 决策。
  • 选择能够容纳 VRAM 的最小 GPU class，然后确认实时库存：使用 
    include:["AVAILABILITY"]
    时必须同时使用 
    product:["SERVERLESS"]
    （不同产品的库存不同）；同一次读取还会给出各数据中心库存，以及 
    cudaVersions
    中哪些 CUDA 版本有可用容量。创建时将 CUDA 下限固定为镜像的 CUDA 版本线（CUDA 12 镜像使用 12.8；只有基于 CUDA 13 构建的镜像才使用 13.0），或者通过 
    gpu.allowedCudaVersions
    指定精确集合。二者互斥（同时发送会返回 400），并且下限版本必须在 
    cudaVersions
    中有库存。绝不能创建到没有库存的 GPU class：scale-to-zero endpoint 如果永远无法获得 worker，看起来已经部署，实际上却永远无法运行。
  • 固定到一张卡。 如果用户需要指定 GPU：选择包含该卡的 pool，并在 
    gpu.excludedTypes
    中列出该 pool 的其他 GPU 类型（
    gpu.pools
    接收的是 pool id，而不是 GPU type id；至少必须保留一种类型）。可在创建时设置，也可事后通过 
    set-endpoint-gpus
    设置。回读结果应列出 pools 和 exclusions，并确认所有其他设置保持不变。
  • 单卡无法容纳模型时使用 Multi-GPU（所需 VRAM 大于有库存的最大单卡）：设置每个 worker 的 GPU 数量大于 1，并在 worker env 中启用 tensor parallelism（例如 vLLM 的 tensor-parallel 设置等于 GPU 数量）。必须说明为什么采用 multi-GPU、其小时费用会按比例增加，并在提交前列出有库存的最大单卡（VRAM、实时库存、$/hr）作为对比。
  • Hub 部署必须固定 pool。 如果省略 
    gpuIds
    ，
    deploy-hub-repo
    会继承 release 的 GPU pool 列表，从而可能悄悄获取顶级显卡，而实际上 24 GB 显卡就能完成任务，其小时费率却高出数倍。除非 release 配置恰好固定了你选择的 pool，否则每次 Hub 部署都必须传入根据你的规格计算得到的 
    gpuIds
    。
  • CPU Serverless。 
    create-endpoint
    只能选择一种硬件：
    gpu
    对象（pools 和 count）或 
    cpu
    数组（根据 
    list-cpu-types
    确定规格）。CPU 工作负载应发送 
    cpu
    ，不能发送 
    gpu
    （不要强行指定 GPU pool），并说明该 CPU flavor 的小时费率。
  • 明确设置所有参数。 REST 创建请求默认使用 FlashBoot 
    OFF
    、
    workers.max
    3 和 
    workers.idleTimeout
    10 秒：必须显式设置 
    flashboot
    和 worker 上下限。始终发送 
    timeout
    （单次请求的执行限制，单位为 ms），并根据最慢的真实 job 设定：v2 创建请求如果不发送该字段，返回结果会包含 
    "timeout": 0
    （已于 2026-10-29 实时验证；为何重要）。Hub 部署后，应回读 
    timeout
    ；如果值为 0，则通过 
    update-endpoint
    设置。账户内所有 
    workers.max
    之和受余额等级配额的限制（初始为 5），并在创建时检查，因此按默认值 3 创建第二个 endpoint 时可能会被拒绝。
  • 说明费用。 创建前引用目录读取结果，说明所选 GPU 的小时费率（乘以 GPU 数量）。创建 endpoint 和运行 smoke test 都会产生费用。

Phase 2 — provision

Phase 2 — 置备

  • Hub release. Deploy its listed release directly (one call). Read the release config first (hardware + env schema) so you pass only env keys it accepts and know its GPU/disk defaults. To reproduce a known-good build, pin the exact release, not "latest".
  • Everything else. Create a template naming the engine image, disk size, exposed ports and env (model id, engine settings, tokens), then the endpoint referencing it. Start scale-to-zero (min workers 0) unless the user asked for warm capacity; never set a nonzero minimum silently: it is an always-on worker that bills around the clock. An image on 
    registry.runpod.net
    is a Hub build: deploy it through 
    deploy-hub-repo
    (the release carries the registry auth); a plain template with that image loops on a registry-auth failure.
  • Provisioning path and cold start. Bake = fastest cold start, biggest image. Download-on-start = small image, slow first request per new worker. Network volume = weights staged once in a GPU-co-located data center and mounted by every worker (writable: coordinate concurrent writes), so new workers skip the download; a prebuilt worker loads from the mount, and your own handler can wrap its load in 
    VolumeCache
    (Runpod Python SDK 
    runpod>=1.12.0
    ) to copy weights to local disk at each cold start (
    runpodctl/reference/model-caching.md
    ). Choose a volume when weights are large (roughly > 30 GB) or shared; size it to the real asset total with headroom, create it in the endpoint's data center (
    runpod-lifecycle-crud
    ), and say which path you took and its cold-start consequence.
  • Weights too large to bake: stage them on a network volume in a data center that has the target GPU, keep minimum workers 0, and prove the wiring with read-backs (for a storage-setup ask those read-backs are the proof; a job on a large model waits for the user's ask); no staging pod the user did not ask for. State both facts plainly: the first worker downloads the weights to the volume once and later cold starts mount that cache, and a volume outside the GPUs' data center cannot mount. Full rule, and the 
    VolumeCache
    variant for the user's own handler: reference/scenarios.md.
  • Multi-region / HA (no auto-sync). Attach one network volume per data center. They do not auto-sync: replicate the data yourself (e.g. 
    aws s3
    sync between the volumes' S3 endpoints), or regions serve stale or missing weights.
  • Required env, gated weights, runtime-optional tokens. Read the release/worker config (
    includeConfig: true
    ) and SET every required env key with no default before deploying (e.g. an embeddings worker's 
    MODEL_NAMES
    ): without its model env a worker loads nothing and 400s every job. When the user named no model for a text-generation worker (a vLLM image with no 
    MODEL_NAME
    ), use the worker's documented default model if its config has one, else the smallest instruct model that fits the card (a 1–2B instruct model on a 16 GB class); state the choice and deploy, don't ask which model. A gated HuggingFace repo whose weights 403 without a token: pass the token in env; if the user gave none, say the download will 403 and ask; never deploy a build that cannot fetch its weights. A token needed only for an optional runtime capability (config default empty) branches on one check you make FIRST: did the user provide a token (pasted, set in the environment, or named as available)?
    • Token provided → using it is the job. Set it as the worker's documented env var at create, say you configured it (by env-var name, never the value), and expect the gated capability to work. Never deploy without wiring a credential the user handed you.
    • No token → deploy anyway. The deploy is NOT conditional on it: deploy, run the job, report what the endpoint returned, then name the token as the unlock.
  • Never print or echo a token (HF, Civitai, registry); reference it as an env var.
  • Hub release。 直接部署列出的 release（一次调用）。先读取 release 配置（硬件和 env schema），以便只传入它接受的 env 键，并了解其 GPU／磁盘默认值。要复现已知可用的构建，应固定准确的 release，而不是使用“latest”。
  • 其他所有来源。 创建一个 template，写明 engine 镜像、磁盘大小、暴露端口和 env（model id、engine 设置、token），然后创建引用该 template 的 endpoint。除非用户要求保留预热容量，否则从 scale-to-zero（min workers 0）启动；绝不能悄悄设置非零最小值，因为这会保持 worker 全天候运行并持续计费。位于 
    registry.runpod.net
    的镜像是 Hub build：应通过 
    deploy-hub-repo
    部署（release 自带 registry 认证）；使用该镜像的普通 template 会因 registry 认证失败而陷入循环。
  • 置备路径与 cold start。 Bake = cold start 最快，但镜像最大。Download-on-start = 镜像较小，但每个新 worker 的首次请求较慢。Network volume = 权重在 GPU 同址的数据中心中暂存一次，并由每个 worker 挂载（可写：必须协调并发写入），因此新 worker 无需再次下载；预构建 worker 从挂载点加载，用户自己的 handler 可以在加载时使用 
    VolumeCache
    （Runpod Python SDK 
    runpod>=1.12.0
    ），在每次 cold start 时将权重复制到本地磁盘（
    runpodctl/reference/model-caching.md
    ）。当权重较大（大约超过 30 GB）或需要共享时，选择 volume；按真实资产总量并留出余量进行容量规划，在 endpoint 所在数据中心创建它（
    runpod-lifecycle-crud
    ），并说明采用的路径及其 cold start 后果。
  • **权重大到无法烘焙进镜像时：**在拥有目标 GPU 的数据中心中，通过 network volume 暂存权重，保持 min workers 为 0，并通过回读证明接线正确（对于存储配置请求，这些回读就是证据；针对大型模型的 job 需要等待用户明确要求）；不得创建用户未要求的 staging pod。必须直接说明两个事实：第一个 worker 会将权重下载到 volume 一次，后续 cold start 则直接挂载该缓存；位于 GPU 数据中心之外的 volume 无法挂载。完整规则及用户自有 handler 的 
    VolumeCache
    变体见 reference/scenarios.md。
  • 多区域／HA（无自动同步）。 每个数据中心分别挂载一个 network volume。它们不会自动同步：必须自行复制数据（例如使用 
    aws s3
    在各 volume 的 S3 endpoint 之间同步），否则各区域可能使用过期或缺失的权重。
  • 必需 env、受限权重、运行时可选 token。 读取 release／worker 配置（
    includeConfig: true
    ），并在部署前设置每个没有默认值的必需 env 键（例如 embeddings worker 的 
    MODEL_NAMES
    ）：缺少 model env 时，worker 不会加载任何内容，每个 job 都会返回 400。如果用户没有为文本生成 worker 指定 model（vLLM 镜像没有 
    MODEL_NAME
    ），且配置中存在文档化默认 model，则使用该默认 model；否则使用能够放入所选显卡的最小 instruct model（16 GB class 上的 1–2B instruct model）；说明所选内容并直接部署，不要询问用户要使用哪个 model。如果某个受限 HuggingFace 仓库的权重在没有 token 时会返回 403：将 token 传入 env；如果用户未提供 token，说明下载将返回 403 并向用户询问；绝不能部署无法获取权重的构建。如果某个 token 仅用于可选的运行时能力（配置默认值为空），则只进行一次最先执行的检查：用户是否提供了 token（直接粘贴、已设置在环境中，或明确表示可用）？
    • 提供了 token → 使用它就是本次任务。 在创建时将其设置到 worker 文档规定的 env 变量中，说明已按 env 变量名完成配置，绝不显示其值，并预期受限能力能够正常工作。用户交付了凭证时，绝不在未连接该凭证的情况下部署。
    • 没有 token → 仍然部署。 部署绝不能以该 token 为条件：完成部署、运行 job、报告 endpoint 返回的内容，然后把该 token 指明为解锁功能所需的凭证。
  • 绝不打印或回显 token（HF、Civitai、registry）；只能以 env 变量名引用。

Phase 3 — warm and smoke-test one real job

Phase 3 — 预热并对一个真实 job 进行 smoke test

A create call that returned an id is not a working endpoint: prove it with one real job first. Block with server-side holds, not a poll loop handed to the user; report once terminal, or with the evidenced report when waiting stops being informative. Silently stopping mid-wait is not an answer.
  • Submit, then wait server-side. Fire one small job (the smallest real input for the modality) with 
    run-endpoint
    , then hold with 
    get-job-status
    wait: 300000
    (the maximum hold). A cold first job outlives 
    runsync
    's hold, whose result is kept only a minute, so 
    runsync-endpoint
    is for a warm endpoint and a fast job. The job proves the worker boots with the real asset set and returns the right output shape.
  • A cold start of 1–5+ minutes on the first job is normal (image pull + model load), not a failure. If the job outlives a hold, hold again on the same job id; do not re-submit (that starts a new job). Between holds check 
    list-endpoint-workers
    : 
    unhealthy
    workers, a job queued beside ready workers, or zero capacity → stop waiting, read the logs. At most three holds per smoke job (
    get-job-status
    wait: 300000
    , same id); after the third, no fourth hold and no tight poll: the evidenced report (endpoint created correctly, job submitted, worker never became ready, what the worker states show) is the complete answer; an endless grind is not.
  • Diagnose a stuck job from the reads you always have: 
    endpoint-health
    , 
    list-endpoint-workers
    , and the 
    workerHealth
    block 
    get-job-status
    attaches on 
    IN_QUEUE
    . 
    • initializing
      on the first job = the cold start may still be running → keep waiting, but read the worker logs (
      source=system
      ) before the next hold: a container crash-looping on an image pull or a CUDA mismatch can still report 
      INITIALIZING
      or 
      RUNNING
      while the job sits in 
      IN_QUEUE
      .
    • unhealthy
      workers, or repeated container starts with no handler output in the logs = a crash loop (a real failure, not slow warmup) → switch the worker image or GPU pool; don't wait it out.
    • total: 0
      for more than a couple of minutes, or 
      throttled
      above 0 = no capacity in the selected pools and data centers → widen them. Prefer a broad, in-stock GPU pool over a scarce high-VRAM tier (e.g. the 48 GB cards) so the endpoint can acquire a worker.
    • A job that never left 
      IN_QUEUE
      was never parsed: no worker read its payload, so NEVER speculate the input was malformed from queue-stuck evidence; report the queue/cold-start fact with the worker states quoted. Never present a cause as "conclusive" that your own reads contradict or don't show: ready workers beside a queued job with nothing in progress point to a broken image; say so from the logs, not from the status alone.
  • Work with the tools you were granted. In an MCP-only session don't reach for a web-fetch tool, shell or browser you don't have to read a worker's input schema or README; it can't be called and the deploy stalls. Take the input shape from 
    list-hub-repos
    includeConfig: true
    and the endpoint's returned error, fix the payload, re-run. Without 
    stream-worker-logs
    , diagnose from 
    endpoint-health
    / 
    list-endpoint-workers
    ; never let an unfinished job hinge on a tool you may not have.
  • On a failure, quote; don't guess. Read the log and quote it; never retry blindly or assert a cause you did not read. A terminal 
    FAILED
    job: report its decisive detail (missing file, gated 403, OOM, wrong-format load error) as written, not what it probably says; fixes are under Error handling below. A terminal job's 
    output
    /
    error
    IS worker-level evidence: quote it verbatim when explaining a failure (a gated-model rejection, a load error); never claim you can't quote evidence while holding a terminal result (no log-stream tool doesn't excuse it). Empty or missing output on COMPLETED jobs from a gated-capability worker is quotable evidence: pair it with the config fact (the documented token env was not set) and state the gate.
创建调用返回 id 并不代表 endpoint 可以工作：必须先用一个真实 job 证明它能够运行。使用 server-side hold 阻塞等待，不要把轮询循环交给用户；到达终态后只报告一次，或者在继续等待已无法提供更多信息时，给出有证据支撑的报告。等待中途静默停止不能算作答案。
  • 提交，然后在服务端等待。 使用 
    run-endpoint
    提交一个小型 job（该 modality 下最小的真实输入），然后通过 
    get-job-status
    和 
    wait:300000
    （最大 hold 时长）保持等待。冷启动的首个 job 会超过 
    runsync
    的 hold 时限，而后者结果只保留一分钟，因此 
    runsync-endpoint
    仅适用于已预热的 endpoint 和快速 job。该 job 证明 worker 能够使用真实资产集合启动，并返回正确的输出结构。
  • 首个 job 经过 1–5 分钟以上的 cold start 属于正常现象（镜像拉取 + 模型加载），并非失败。如果 job 超过一次 hold 的时长，应使用同一个 job id 再次 hold；不要重新提交（那会启动新 job）。两次 hold 之间检查 
    list-endpoint-workers
    ：如果存在 
    unhealthy
    worker、job 在 ready worker 旁仍处于排队状态，或者容量为零，应停止等待并读取日志。每个 smoke job 最多执行三次 hold（使用 
    get-job-status
    和 
    wait:300000
    ，且保持同一个 id）；第三次之后不得执行第四次 hold，也不得进行紧密轮询。此时，有证据支撑的报告（endpoint 已正确创建、job 已提交、worker 从未变为 ready、worker 状态所显示的情况）就是完整答案；无休止地消耗资源不是答案。
  • 根据始终可用的读取结果诊断卡住的 job：
    endpoint-health
    、
    list-endpoint-workers
    ，以及 
    get-job-status
    在 
    IN_QUEUE
    状态下附加的 
    workerHealth
    信息。
    • 首个 job 显示 
      initializing
      ，说明 cold start 可能仍在运行 → 继续等待，但下一次 hold 前先读取 worker 日志（
      source=system
      ）：因镜像拉取或 CUDA 不匹配而不断崩溃的容器，仍可能报告 
      INITIALIZING
      或 
      RUNNING
      ，同时 job 一直处于 
      IN_QUEUE
      。
    • worker 处于 
      unhealthy
      状态，或容器反复启动且日志中没有任何 handler 输出 = 崩溃循环（这是真实故障，并非缓慢预热）→ 更换 worker 镜像或 GPU pool；不要继续干等。
    • total: 0
      持续超过几分钟，或 
      throttled
      大于 0 = 所选 pools 和数据中心没有容量 → 扩大范围。相比稀缺的高 VRAM 档位（例如 48 GB 显卡），优先选择范围较广且有库存的 GPU pool，使 endpoint 能够获得 worker。
    • job 从未离开 
      IN_QUEUE
      ，说明它从未被解析：没有 worker 读取其 payload。因此，绝不能仅凭任务卡在队列中的证据推测输入格式错误；应报告队列／cold start 的事实，并引用 worker 状态。ready worker 与排队 job 同时存在、却没有任何处理活动时，指向镜像损坏；这一点必须依据日志说明，而不能只依据状态推断。
  • 使用已授予的工具。 在 MCP-only 会话中，不要尝试调用当前未授予的 web-fetch tool、shell 或 browser 去读取 worker 的输入 schema 或 README；这些工具无法调用，只会让部署停滞。输入结构应取自 
    list-hub-repos
    的 
    includeConfig: true
    结果以及 endpoint 返回的错误；修复 payload 后重新运行。如果没有 
    stream-worker-logs
    ，则使用 
    endpoint-health
    ／
    list-endpoint-workers
    进行诊断；绝不能让未完成的 job 依赖一个可能没有的工具。
  • 失败时引用证据，不要猜测。 读取日志并直接引用；绝不盲目重试，也绝不能断言一个未经读取的原因。job 终态为 
    FAILED
    时：按原文报告决定性细节（文件缺失、受限访问 403、OOM、格式错误的加载错误），而不是转述你认为可能的原因。终态 job 的 
    output
    ／
    error
    就是 worker 层证据：解释失败时必须逐字引用。已完成但输出为空或缺失的受限能力 worker job 同样具有可引用证据：结合配置事实（文档规定的 token env 未设置）说明该限制。

Phase 4 — hand back

Phase 4 — 交回结果

  • Read back what was asked. When the user asked for specific behavior settings (warm/minimum workers, idle timeout, FlashBoot, a GPU pin), read the endpoint back after the create/update and confirm those exact fields landed, plus 
    timeout
    : an unread write is an unverified write, and the write call's own response is not a read. The REST PATCH is sparse (omitted fields are left untouched) except for the GPU selection: 
    gpu.pools
    sent without 
    gpu.excludedTypes
    clears the exclusions, so send both, and after a pin confirm the scaling fields too. The read-back result goes in the final answer.
  • The hand-back is a complete final summary the user never scrolls back from: endpoint name and id; worker image or Hub repo, why that engine, and the read it came from; GPU pool and hourly price; smoke job id and terminal outcome with proof (output reference, quoted failure, or, if the wait ended without a terminal job, the evidenced report: job id, submitted payload, worker states observed, recommended next step); current worker state; sync and async URLs; kept (and what now bills) or deleted (read-back confirmed); a ready-to-run 
    curl
    using 
    $RUNPOD_API_KEY
    (never the literal key); the sizing arithmetic, format finding, and caching/placement facts.
  • Cleanup. Default to KEEP what the user asked for: say exactly what is now billing (the endpoint, any min-worker, any network volume) and how to delete it later. Tear down (endpoint, its template, any network volume, each confirmed gone by a read-back) only when the user framed the journey as a throwaway test or asks for cleanup.
  • 回读用户要求的内容。 如果用户要求特定的行为设置（预热／最小 worker 数、idle timeout、FlashBoot、固定 GPU），在创建／更新后回读 endpoint，确认这些字段和 
    timeout
    已准确生效。未回读的写入就是未经验证的写入，而写调用的响应本身不算回读。REST PATCH 采用稀疏更新（省略的字段保持不变），GPU 选择除外：只发送 
    gpu.pools
    而不发送 
    gpu.excludedTypes
    会清空排除项，因此两者必须一起发送；固定 GPU 后，还要确认扩缩容相关字段。最终答案中应包含回读结果。
  • 交付内容。 最终总结必须完整，使用户无需回翻上下文：endpoint 名称和 id；worker 镜像或 Hub 仓库、选择该 engine 的原因及其依据来源；GPU pool 和小时费率；smoke job id 和终态结果及证据（输出引用、引用的失败信息，或等待结束但尚未进入终态时的有证据报告：job id、提交的 payload、观察到的 worker 状态及建议的下一步）；当前 worker 状态；同步和异步 URL；保留（以及当前哪些资源会计费）或删除（并通过回读确认）；一条使用 
    $RUNPOD_API_KEY
    的即用型 
    curl
    （绝不能使用实际密钥）；规格计算过程、格式检测结果，以及缓存／部署位置相关事实。
  • 清理。 默认保留用户要求部署的内容：准确说明当前哪些资源会计费（endpoint、任何 min-worker、任何 network volume），以及后续如何删除。只有当用户明确表示这是一次 throwaway test 或要求清理时，才拆除 endpoint、对应 template 和任何 network volume，并通过回读逐一确认它们已经删除。

Hard rules

硬性规则

  • Deploy only the source the user named. Never create an endpoint they did not ask for, and never touch or delete another endpoint, template, or volume.
  • Never invent a model URL, a community template repo, or a Docker image. If the source can't be resolved from what the user gave plus the catalog, ask once or say "none found" and stop.
  • Also binding: every rule in phases 0–4 above.
  • Cold-start asks on a new endpoint (an existing endpoint is runpod-endpoint-ops'): read reference/scenarios.md first. Create with the levers that add no standing charge, and set a minimum worker only when asked or accepted: it bills around the clock.
  • 只部署用户指定的来源。绝不能创建用户未要求的 endpoint，也绝不能操作或删除其他 endpoint、template 或 volume。
  • 绝不能虚构 model URL、社区模板仓库或 Docker 镜像。如果无法根据用户提供的信息和目录解析出来源，应询问一次，或者回答“none found”并停止。
  • 以下规则同样具有约束力：Phase 0–4 中的每一条规则。
  • 新 endpoint 的 cold-start 请求（现有 endpoint 属于 
    runpod-endpoint-ops'
    ）：先阅读 reference/scenarios.md。创建时采用不会产生持续费用的调整手段；只有在用户明确要求或接受时，才设置 min worker，因为它会全天候计费。

Error handling

错误处理

  • Gated / 
    403
    → say so and ask for the token; don't auto-recreate. OOM → up one GPU class (quote the new hourly cost); still failing after one step-up → re-examine the engine choice (switch image in-turn), not more GPU or waiting.
  • Payload 
    400
    → fix the payload or the missing env from the release config and re-run; don't stop to ask. Create 
    409
    (name collision) → fresh short suffix, retry once, then surface it and stop. A 
    204
    delete with no body is success; confirm with a read-back.
The full failure → action table (also a stuck queue past a hold, token-gated empty output, GGUF on vLLM, an empty catalog) is in reference/error-handling.md (guide id 
runpod-serverless-deploy/error-handling
). Read it when a create or the smoke job fails, before you retry or report a cause.
受限访问／403 → 明确说明并向用户询问 token；不要自动重新创建。OOM → 提升一个 GPU class（说明新的小时费率）；提升一级后仍然失败，则重新评估 engine 选择（当场切换镜像），而不是继续增加 GPU 或等待。Payload 400 → 根据 release 配置修复 payload 或缺失的 env，然后重新运行；不要停下来询问。创建返回 409（名称冲突）→ 使用新的短后缀重试一次，然后如实报告并停止。删除返回 204 且没有 body 表示成功；必须通过回读确认。
完整的“失败 → 动作”对照表（还涵盖 hold 结束后任务仍在队列中、token 受限导致的空输出、GGUF 用于 vLLM、目录为空等情况）见 reference/error-handling.md（guide id 
runpod-serverless-deploy/error-handling
）。创建或 smoke job 失败时，应在重试或报告原因之前阅读该文档。

Contract

Contract

The cross-journey answer contract (definite facts from reads, commit-don't-hedge, mutations bind to this conversation's creations, a standalone final message, honest failure, granted tools only) is defined in the 
runpod-mcp
skill and applies to every reply from this journey. If the 
runpod-mcp
skill has not been loaded in this conversation, load it now, before the next tool call: its rules on which resources you may change, when a request is the go, and when to stop for the user are not repeated here.
跨流程答案 contract（只陈述从读取结果中得出的确定事实、果断执行而不含糊、修改操作仅限本次对话中创建的资源、提供独立完整的最终消息、诚实报告失败、只使用已授予的工具）由 
runpod-mcp
skill 定义，并适用于本流程中的每一条回复。如果本次对话尚未加载 
runpod-mcp
skill，应在下一次 tool 调用前立即加载：其中关于可以修改哪些资源、何时用户请求构成执行授权，以及何时应停下来询问用户的规则，并未在此重复。