runpod-endpoint-ops
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseEndpoint ops
Endpoint 运维
You operate a Serverless endpoint that already exists. Everything starts by reading its current state (the endpoint config, its health, its workers, its logs) before you say anything or change anything. Diagnosis is always read-only. You change a setting only when the user asked for a change, and then only the exact setting agreed.
One property of the Runpod update path drives the central rule here: an endpoint update is a PATCH. Only the fields present in the body change and omitted fields are left untouched, so you send exactly the setting you were asked to change and nothing else. The one exception is the GPU selection: and are a single selection, so a PATCH that sends without clears the exclusions. When you touch either, send both. What the PATCH does not give you is proof: its own response is not a read. Read the config first for the before value, and read it back afterward for the after value.
gpu.poolsgpu.excludedTypespoolsexcludedTypes你负责操作一个已经存在的 Serverless endpoint。所有操作都必须先读取其当前状态(endpoint 配置、健康状态、workers 和日志),然后才能给出结论或进行更改。诊断始终是只读的。只有当用户要求更改时,你才修改设置,而且只修改双方约定的准确设置。
Runpod 更新路径的一项特性决定了本流程的核心规则:endpoint 更新采用 PATCH。请求体中只有出现的字段会发生变化,省略的字段保持不变,因此你只应发送用户要求更改的设置,不附带其他内容。唯一的例外是 GPU 选择: 和 共同构成一个选择,所以仅发送 而不发送 的 PATCH 会清空 exclusions。修改其中任何一个时,必须将两者一起发送。不过,PATCH 本身无法提供证明:更新响应本身不算一次读取。先读取配置以获得变更前的值,更新后再读取以获得变更后的值。
gpu.poolsgpu.excludedTypespoolsexcludedTypesTool binding
工具绑定
This skill reads before it mutates, and mutates only what the user authorized. If a tool named here is missing from the session's tool list, say so and use the nearest read instead of inventing a result.
| Capability | Tool |
|---|---|
| List / inspect endpoints: resolve "which endpoint" and read the full current config | |
| Endpoint health: the stuck-queue / dying-worker evidence | |
| List workers (per-worker detail) | |
| Read a worker's logs: the crash / model-load failure line | |
| GPU stock per data center: is a stuck queue a capacity problem? | |
| CUDA versions with capacity, per GPU type | the same read ( |
| Update endpoint settings (worker min/max, idle timeout, scaler, FlashBoot, disk, env): the gated tune | |
Pin GPUs / set GPU count ( | |
| Release history: which build the workers are running | |
| Cancel one job (gated) | |
| Purge the queue (gated) | |
| Submit / poll a job to re-verify after a change | |
此 skill 会先读取再执行变更,并且只变更用户授权的内容。如果当前 session 的工具列表中缺少此处指定的某个工具,应明确说明,并改用最接近的读取工具,不得虚构结果。
| 能力 | 工具 |
|---|---|
| 列出或检查 endpoints:确定“具体是哪个 endpoint”,并读取完整的当前配置 | |
| Endpoint 健康状态:queue 卡住或 workers 宕机的证据 | |
| 列出 workers(每个 worker 的详细信息) | |
| 读取 worker 日志:崩溃或模型加载失败的关键行 | |
| 每个数据中心的 GPU 库存:判断 queue 卡住是否由容量问题导致 | |
按 GPU 类型查看有余量的 CUDA 版本:使用同一次读取( | 同一次读取 |
| 更新 endpoint 设置(worker min/max、idle timeout、scaler、FlashBoot、disk、env):受控调优 | |
固定 GPUs 或设置 GPU 数量( | |
| Release 历史:确认 workers 正在运行哪个 build | |
| 取消一个 job(受控操作) | |
| 清空 queue(受控操作) | |
| 提交或轮询 job,以便变更后重新验证 | |
Phase 1 — diagnose (always, read-only)
第 1 阶段——诊断(始终只读)
Resolve the endpoint (list endpoints and match by name if needed), read its config, then read health and workers before forming a conclusion. Classify against the symptom:
- Jobs stuck . Read health and the worker list. If max workers is 0 or all workers are throttled/unhealthy, the queue has nothing to run on. An endpoint idle for 3 days has
IN_QUEUElowered to 2 and after 7 days to 0, and it stays there until raised, so a long-idle endpoint "stuck" on its first job is usually this. Separate the causes: worker limit (max-workers too low or lowered by idleness, or min 0 with a cold scaler) vs capacity (workers can't acquire the selected GPU:workers.maxabove 0; cross-check that GPU type's availability withthrottled, and its CUDA versions with capacity when the endpoint pins a CUDA floor) vs crash loop (workers cycling: read their logs withproduct:["SERVERLESS"]; a container that exits before the handler runs can still report RUNNING or INITIALIZING while the job stays queued, so the logs settle it, not the worker status). Name which one, quoting the health/worker numbers you read.source=system - Workers crash-looping. List workers, find the failing ones, read their logs, and quote the decisive line: a missing model file, a gated 403, an OOM, a bad handler import. Report what the log says, not a guess. OOM → the GPU is undersized for the real memory pattern; a load/import error → the image or env is wrong.
- Cold starts hurt. Read the config. The levers, in order: enable FlashBoot; raise the idle timeout so warm workers linger between bursts; set a nonzero min-worker to keep capacity warm (state that a min-worker bills around the clock; it is an always-on worker); move the model load off the boot with model caching (network volume, Runpod's model repository, or weights baked into the image; on a volume-backed endpoint running the user's own handler, copies the weights to local disk before the load); and make sure the scaler isn't scaling to zero between every request. Name the levers and their cost before touching anything.
VolumeCache - Load-balancer endpoint, workers never ready. On a load-balancer (custom-HTTP) endpoint the load balancer decides readiness by polling the worker's health endpoint: (default
HEALTH_CHECK_PATH) on/ping(default 80), which is separate from the servingPORT_HEALTH(default 80). Workers stuck un-ready while requests arrive usually means nothing answers that path on that port; the endpoint read gives you the exact health URL being polled. The fix is a redeploy that exposes the port and serves the health path (seePORT), not a scaler change.runpod-serverless-deploy
End phase 1 with the diagnosis and the recommended change, but do not apply it unless the prompt authorized a change. If the ask was only "why / what's wrong", you are done here.
先确定 endpoint(必要时列出 endpoints 并按名称匹配),读取其配置,再读取健康状态和 workers,然后才能形成结论。根据症状进行分类:
- Jobs 卡在 。 读取健康状态和 worker 列表。如果 max workers 为 0,或所有 workers 都被 throttled 或处于不健康状态,queue 就没有可运行 job 的资源。闲置 3 天的 endpoint,其
IN_QUEUE会被降至 2;闲置 7 天后会降至 0,并会一直保持,直到手动调高。因此,长时间闲置的 endpoint 在第一个 job 上“卡住”通常就是由此导致。应区分不同原因:worker 限制(max workers 过低或因闲置而被降低,或者 min 为 0 且 scaler 为冷启动)、容量(workers 无法获取所选 GPU:workers.max大于 0;使用throttled交叉检查该 GPU 类型的 availability;当 endpoint 固定了 CUDA 下限时,还要核对其有余量的 CUDA 版本)或崩溃循环(workers 不断循环:使用product:["SERVERLESS"]读取日志;在 handler 运行前退出的 container 即使 job 仍在排队,也可能显示 RUNNING 或 INITIALIZING,所以应以日志而非 worker 状态为准)。明确指出属于哪一种,并引用你读取到的健康状态或 worker 数值。source=system - Workers 持续崩溃循环。 列出 workers,找出故障 workers,读取日志,并引用决定性的日志行:模型文件缺失、受访问控制的 403、OOM 或 handler import 错误。报告日志明确说明的内容,不要猜测。OOM → GPU 相对于真实的内存使用模式过小;加载或 import 错误 → image 或 env 有误。
- 冷启动影响性能。 读取配置。可调项按优先顺序如下:启用 FlashBoot;提高 idle timeout,让 warm workers 在流量峰值之间继续保留;设置非零 min-worker,以保持容量常驻(说明 min-worker 会全天候计费;它是 always-on worker);通过 model caching 将模型加载移出启动流程(network volume、Runpod 的 model repository,或将 weights 预置到 image 中;对于运行用户自有 handler 的 volume-backed endpoint,会在加载前将 weights 复制到本地 disk);并确保 scaler 不会在每次请求之间都缩容到零。在进行任何更改前,先说明这些可调项及其成本。
VolumeCache - Load-balancer endpoint,workers 始终无法就绪。 在 load-balancer(custom-HTTP)endpoint 上,load balancer 通过轮询 worker 的 health endpoint 判定 readiness:(默认
HEALTH_CHECK_PATH),位于/ping(默认 80)上;该端口独立于提供服务的PORT_HEALTH(默认 80)。当请求已经到达而 workers 仍卡在未就绪状态时,通常意味着该端口上的对应路径没有响应;读取 endpoint 可获得实际轮询的准确 health URL。修复方式是重新部署,暴露相应端口并提供 health path(参见PORT),而不是调整 scaler。runpod-serverless-deploy
第 1 阶段结束时给出诊断和建议的更改,但除非 prompt 明确授权更改,否则不要执行。如果用户只问“为什么”或“出了什么问题”,到此即可。
Phase 2 — tune safely (only if the user authorized a change)
第 2 阶段——安全调优(仅在用户授权更改时执行)
Advance only when the prompt asks to change a named endpoint's setting.
- The read-back invariant. Read the endpoint's full config first, so you hold the before values. Apply the change as a PATCH carrying only the fields the user authorized. Sending fields you were not asked about is how you overwrite a setting they tuned by hand. Then read the config back and confirm the target field holds its new value and the neighbours you quoted are unchanged.
- Autoscaling tune, bursty traffic. For bursty workloads, tune the scaler and worker band: raise max workers for the burst ceiling, set the queue-delay or request-count scaler target to how fast you want to absorb the burst, and set idle timeout to how long to hold warm workers between waves (a queue endpoint scaling on request count rejects , and the account's worker quota can refuse a higher
workers.idleTimeout). Change only the fields the user agreed; state the cost of a higher min-worker.workers.max - Pin to one GPU type. When the user wants the endpoint locked to one exact card, select the pool that contains the card and list the pool's other GPU types in (at least one type must remain), with the CUDA floor if given.
gpu.excludedTypesbuilds that exclusion list for you and re-sends the endpoint's other settings unchanged;set-endpoint-gpuscan carry the sameupdate-endpointobject, but theregpuandpoolsare one selection, so send both together.excludedTypesandallowedCudaVersionsare mutually exclusive (400 if both are sent). Changing the CUDA constraint or the pool does not evict running workers; bounceminCudaVersionto 0 and back when the change must apply now. Read the endpoint back after a pin and confirm the scaling settings survived.workers.max
只有当 prompt 要求更改某个已明确指定的 endpoint 设置时,才能继续。
- 回读校验原则。 先读取 endpoint 的完整配置,保留变更前的值。以 PATCH 应用更改,请求体只携带用户授权的字段。发送用户未要求的字段,会覆盖他们手动调优的设置。然后回读配置,确认目标字段保持新值,且你引用的相邻设置未发生变化。
- 面向突发流量的 autoscaling 调优。 对于突发型工作负载,调整 scaler 和 worker 区间:提高 max workers,作为突发流量上限;将 queue-delay 或 request-count scaler target 设为你希望吸收突发流量的速度;将 idle timeout 设为你希望在多轮流量之间保留 warm workers 的时长(基于 request count 扩缩容的 queue endpoint 会拒绝 ,账户的 worker quota 也可能拒绝更高的
workers.idleTimeout)。只更改用户同意的字段;说明更高 min-worker 的成本。workers.max - 固定到一种 GPU 类型。 当用户希望将 endpoint 锁定到某张确切的卡时,选择包含该卡的 pool,并将 pool 中其他 GPU 类型列在 中(至少保留一种类型),如果用户指定了 CUDA 下限,也一并设置。
gpu.excludedTypes会为你构建 exclusion list,并原样重新发送 endpoint 的其他设置;set-endpoint-gpus也可以携带相同的update-endpoint对象,但其中gpu和pools构成一个选择,因此必须一起发送。excludedTypes与allowedCudaVersions互斥(同时发送会返回 400)。更改 CUDA constraint 或 pool 不会驱逐正在运行的 workers;当必须立即应用更改时,将minCudaVersion先调至 0,再调回。固定后回读 endpoint,确认 scaling 设置仍然保留。workers.max
Phase 3 — manage the queue (gated actions)
第 3 阶段——管理 queue(受控操作)
- Cancel one job. When the user names a job to cancel, cancel that job by id. Cancel affects only queued/in-progress jobs. After the cancel, VERIFY with a fresh read. The cancel call's own response is not verification (an unread write is an unverified write); the final answer reports the re-read state. If the re-read shows the job already reached a terminal state before your cancel, say exactly that; it is a complete, honest outcome.
get-job-status - Purge the queue. When the user wants every pending job cleared, purge the queue. This removes all pending jobs; in-progress jobs keep running. Say clearly that it clears the whole pending queue, not just the bad ones, and do it only on an explicit "purge / clear the queue". Re-read health afterward to confirm the queue drained.
- 取消一个 job。 当用户指定要取消的 job 时,按 id 取消。取消仅影响 queued 或 in-progress jobs。取消后,使用全新的 读取进行 VERIFY。取消调用自身的响应不算验证(未回读的写入就是未经验证的写入);最终答复应报告重新读取后的状态。如果回读显示该 job 在取消前已经进入 terminal state,就准确说明这一点;这就是完整且诚实的结果。
get-job-status - 清空 queue。 当用户希望清除所有 pending jobs 时,清空 queue。此操作会删除所有 pending jobs;in-progress jobs 会继续运行。明确说明它会清除整个 pending queue,而不只是有问题的 jobs,并且仅在用户明确表示“purge / clear the queue”时执行。操作后重新读取健康状态,确认 queue 已排空。
Hard rules
硬性规则
- Every settings change is read → update → read: (1) read the current config, (2) apply the update, (3) a FRESH read AFTER it. The before/after values the final answer quotes come from reads (1) and (3). The update call's own response is not a read; never claim a read-back you did not perform.
get-endpoint - Change only the setting the user authorized, on the endpoint they named. Never tune, pin, or purge an endpoint the user did not name.
- Diagnosis is read-only, and read-only includes ZERO job submissions. Never fire a /
run"test probe" while diagnosing: a submitted job bills compute and changes the queue, so it is a mutation, and on a broken endpoint it burns money to reproduce what the worker states already show. Diagnose from the endpoint/health/worker/log reads only; a recommended fix is presented, not applied, unless the prompt authorizes it.runsync - A queue purge clears the entire pending queue. Say so before doing it; never purge on a vague "clean this up".
- State the cost of any setting that bills continuously (a nonzero min-worker keeps workers warm and billed) before applying it.
- Quote the health/worker/log field you read. No "probably" about a number the tool returns.
- Never cancel or purge jobs the user did not ask you to; do not "helpfully" clear a queue during a diagnosis.
- If a read (health, workers, logs) fails, report it and stop; do not guess the cause.
- 所有设置变更都必须遵循读取 → 更新 → 读取:(1) 读取当前配置,(2) 应用更新,(3) 在之后执行一次全新的 读取。最终答复中引用的变更前和变更后数值来自读取 (1) 和 (3)。更新调用自身的响应不算读取;绝不能声称执行了实际未完成的回读。
get-endpoint - 只更改用户授权的设置,并且只能操作用户指定的 endpoint。绝不调整、固定或清空用户未指定的 endpoint。
- 诊断过程只读,且只读包括 ZERO job 提交。诊断期间绝不能触发 或
run“测试探针”:提交 job 会产生计算费用并改变 queue,因此在故障 endpoint 上复现 worker 状态已经显示的问题只会白白消耗费用。诊断只能依据 endpoint、health、worker 和 log 的读取结果;推荐修复方案只提出,不执行,除非 prompt 明确授权。runsync - 清空 queue 会清除整个 pending queue。执行前必须明确说明这一点;绝不能因为用户含糊地说“清理一下”就执行。
- 在应用任何会持续计费的设置之前,先说明其成本(非零 min-worker 会让 workers 保持 warm 并持续计费)。
- 引用你实际读取到的 health、worker 或 log 字段。对于工具返回的数值,不要使用“可能”等含糊表述。
- 绝不取消或清空用户未要求处理的 jobs;诊断期间不要“顺手”清空 queue。
- 如果读取操作(health、workers、logs)失败,应报告错误并停止;不得猜测原因。
Error handling
错误处理
- A setting nobody touched comes back changed → it was not the PATCH, which leaves omitted fields untouched (except sent without
gpu.pools, which clears the exclusions). Look at the other write paths: a GPU pin that re-sent the whole config, a template save, a redeploy, or a concurrent change by someone else. Re-read before re-applying anything.excludedTypes - with ready workers and nothing in progress → do not wait it out: read the worker logs (
IN_QUEUE); a container that exits before the handler runs looks exactly like this and points at a broken image, not at the scaler.source=system - Worker logs show OOM → the GPU is undersized; the fix is a bigger GPU (re-deploy via ), not a scaler tweak.
runpod-serverless-deploy - Cancel/purge returns success but health still shows queued jobs → re-read after a moment; the queue count settles slightly after the action.
- answers 404 for a job id that was valid → the job's TTL (24 h from submission by default) expired and removed it; it is not a wrong id.
get-job-status - A GPU-pin request when is missing from the session → make the same selection through
set-endpoint-gpus, sendingupdate-endpointandgpu.poolstogether.gpu.excludedTypes
- 某个用户并未要求改动的设置在操作后发生了变化 → 这不是 PATCH 所致,因为 PATCH 会保持省略字段不变(唯一例外是发送 而未发送
gpu.pools,这会清空 exclusions)。检查其他写入路径:重新发送完整 config 的 GPU pin、template 保存、redeploy,或其他人的并发更改。重新应用任何内容之前,先再次读取。gpu.excludedTypes - 、workers 已就绪且没有任务正在执行 → 不要等待其自行恢复:读取 worker 日志(
IN_QUEUE);在 handler 运行前退出的 container 正是这种表现,指向有问题的 image,而不是 scaler。source=system - Worker 日志显示 OOM → GPU 规格过小;修复方式是使用更大的 GPU(通过 重新部署),而不是调整 scaler。
runpod-serverless-deploy - Cancel 或 purge 返回成功,但健康状态仍显示 queued jobs → 稍后重新读取;操作后 queue count 需要一点时间才会稳定。
- 对原本有效的 job id 返回 404 → 该 job 的 TTL(默认从提交起 24 小时)已过期并被删除;并不是 id 错误。
get-job-status - 当 session 中缺少 时发起 GPU-pin 请求 → 通过
set-endpoint-gpus执行相同选择,并一起发送update-endpoint和gpu.pools。gpu.excludedTypes
Contract
契约
The cross-journey answer contract (definite facts from reads, commit-don't-hedge, mutations bind to this conversation's creations, a standalone final message, honest failure, granted tools only) is defined in the skill and applies to every reply from this journey. If the skill has not been loaded in this conversation, load it now, before the next tool call: its rules on which resources you may change, when a request is the go, and when to stop for the user are not repeated here.
runpod-mcprunpod-mcp跨流程回答契约(依据读取结果给出确定事实、commit-don't-hedge、写操作只能作用于本对话中创建的资源、单独输出 final message、如实报告失败、仅使用已授权 tools)由 skill 定义,并适用于本流程的每次回复。如果当前对话尚未加载 skill,请在下次 tool 调用前立即加载:其中关于可以更改哪些资源、何时用户的请求即为执行许可,以及何时应停止并询问用户的规则未在此重复。
runpod-mcprunpod-mcp