baseten
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseBaseten Product Overview
Baseten产品概述
Production AI inference platform - serve and scale open-source, custom, and fine-tuned models with the fastest runtimes,
cross-cloud HA, and seamless developer workflows.
- Dedicated Inference - deploy any model, performance-optimized + horizontally scaled. Authored as auto-wrapped Truss server, custom Docker server, or compound/orchestrated deployment via Chains.
- Model APIs - pre-optimized hosted APIs for popular models. Path to graduate to dedicated.
- Training - two paths: Truss Train (BYO container, any framework, full hardware control) and Loops (Tinker-compatible managed SDK for SFT + async RL; paired trainer + sampling server, live weight transfers, one-click checkpoint deploy). Multi-node, 1T+ params, 10TB+ datasets, H100/H200/B200. Remote access: SSH and VS Code/Cursor tunnels into containers.
- Frontier Gateway - operate your own foundation model B2C.
生产级AI推理平台——凭借最快的运行时、跨云高可用(HA)以及流畅的开发者工作流,部署并扩展开源、自定义和微调模型。
- 专属推理 - 部署任意模型,性能优化+水平扩展。可通过自动封装的Truss服务器、自定义Docker服务器,或是借助Chains实现复合/编排式部署。
- 模型API - 针对热门模型预优化的托管API。可升级至专属部署。
- 训练 - 两种路径:Truss Train(自带容器,支持任意框架,完全硬件控制)和Loops(兼容Tinker的托管SDK,用于SFT+异步RL;配对训练器+采样服务器,实时权重传输,一键部署检查点)。支持多节点、1T+参数、10TB+数据集,适配H100/H200/B200硬件。远程访问:通过SSH和VS Code/Cursor隧道连接容器。
- 前沿网关(Frontier Gateway) - 运营您自己的大模型B2C业务。
Agent DX Toolkit
Agent DX工具包
| Component | Provides | Install |
|---|---|---|
| Interact with backend (~REST API, CRUD): models, deployments, training, environments, secrets, chains. API-key auth. | |
| Semantic search + filesystem of | |
| Needed for model/chain push from local code, watch (= live patch). Needs | |
| | reachable via HTTP |
| This skill | | |
| 组件 | 功能 | 安装方式 |
|---|---|---|
| 与后端交互(类REST API,CRUD操作):模型、部署、训练、环境、密钥、Chains。API密钥认证。 | |
| | |
| 用于从本地代码推送模型/Chain、实时补丁(watch)。需执行一次 | |
| | 可通过HTTP访问 |
| 本技能 | 按需加载 | |
Setup
安装设置
- Any subset works, full install recommended.
- Suggest additional installs when the current task benefits from or requires them; help user with installation, but elicit preferences first.
- Ensure is provided when installing Baseten MCP (user can create key at
BASETEN_MCP_KEY). Caveat: an MCP instance binds to one org/workspace at install time; switching the bound workspace later is not supported. To work with multiple workspaces, install additional MCP instances under different names with different keys (see last bullet of this section).app.baseten.co/settings/api_keys - Truss CLI only needed for making deployments (check ); prior login (multi-workspace users must provide
truss --version). Explore with--remote <name>.truss [subcommand] --help - Docs MCP missing → grep / fetch .
llms.txt - Backend MCP is API-key-only (currently); OAuth-only harnesses can still use the other components.
- If backend MCP server is needed for different orgs/workspaces, add multiple MCP instances with different names/keys or use env-var expansion in the agent's config file and set the env var to the respective workspace's key.
- 可选择安装任意子集,推荐完整安装。
- 当当前任务需要或受益于额外组件时,建议用户安装;协助用户安装,但需先询问用户偏好。
- 安装Baseten MCP时需提供(用户可在
BASETEN_MCP_KEY创建密钥)。注意:MCP实例在安装时绑定到一个组织/工作区;后续不支持切换绑定的工作区。如需在多个工作区操作,请使用不同密钥安装多个同名不同配置的MCP实例(见本节最后一点)。app.baseten.co/settings/api_keys - Truss CLI仅在创建部署时需要(检查);首次登录(多工作区用户需指定
truss --version)。可通过--remote <name>查看详情。truss [子命令] --help - 若缺少Docs MCP → 搜索/获取。
llms.txt - 后端MCP目前仅支持API密钥认证;仅支持OAuth的工具仍可使用其他组件。
- 若需为不同组织/工作区使用后端MCP服务器,可添加多个不同名称/密钥的MCP实例,或在代理配置文件中使用环境变量扩展,并将环境变量设置为对应工作区的密钥。
Pick your authoring surface (for creating deployments)
选择部署创作方式
First-pass decision. Many real workloads blend rows — treat this as a starting point, not a rule. When unsure, sketch
the IO shape and per-step hardware needs before picking.
Surfaces are stacked by opinion-strength, not just author convenience: engines (TRT-LLM, BEI, BIS-LLM) ship
performance-tuned for one architecture and are the fastest path when they fit; custom Docker servers wrap mature
inference servers (vLLM, SGLang, TGI, Triton, NIM); Python Truss is the escape hatch for arbitrary code in the
request path; Chains add typed inter-step transport with built-in rate limiting, connection management, structured
error propagation, and binary IO — features you'd otherwise rebuild around N raw Trusses. Python Truss and Chains share
live-patch iteration (); all flavors support per-replica autoscaling, scale-to-zero, and environments /
promotions.
truss watch| You want… | Flavor | Specialization | When NOT to pick |
|---|---|---|---|
| Hosted LLM, no deploy step | Model APIs | | model not in catalog; need custom hardware, requirements, stability... |
| LLM on an off-the-shelf server (vLLM / SGLang / TGI / Triton / NIM) | Custom Docker server | | the server doesn't exist or you need Python in the request path |
| LLM/embedding via a Baseten engine (TRT-LLM / BEI / BIS-LLM); minimal config, no Python | Engine-only | | architecture not covered by an engine; you need custom logic |
| Custom Python in the request path (pre/post, custom arch, weird IO) | Python-class Truss ( | | an engine or off-the-shelf server fits — pick that, it's faster to ship |
| Multi-step pipeline with heterogeneous hardware / per-step scaling (RAG, ASR→LLM→TTS, fan-out, chunking) | Chains | | one-stage or homogeneous — a single Truss is simpler |
Orthogonal operational surfaces (independent of which flavor above):
- Iterate / patch a deployment →
model-dev-loop.md - Promote, environments, autoscaling →
deployment-lifecycle.md - Gradual traffic shift to new deployment (pause/resume/cancel) → rolling deployments:
deployment/rolling-deployments.mdx - Call a deployment → (custom) or
inference-api.md(hosted)model-apis.md - High-throughput batch calls (embeddings, reranking) → Performance client (Rust, 1200+ req/s):
inference/performance-client.mdx - Programmatic control plane →
management-api.md
Real-world nuances the table can't capture:
- Hybrids exist. A can wrap an engine for pre/post-processing; a Chain entrypoint can be a Python class while internal Chainlets use engines.
model.py - Chain websockets are entrypoint-only. Intra-chainlet calls only stream output, but bi-di usually not needed on those edges.
- Engine performance vs flexibility. TRT-LLM is the fastest path for many LLMs but its config surface is opaque. Worth the trade only when latency/throughput is a real constraint.
初步决策。许多实际工作负载会混合多种方式——将此视为起点,而非规则。不确定时,先勾勒IO形态和每一步的硬件需求再做选择。
各种方式按推荐优先级排序,而非仅考虑作者便利性:引擎(TRT-LLM、BEI、BIS-LLM)针对特定架构进行性能调优,适配时是最快路径;自定义Docker服务器封装成熟的推理服务器(vLLM、SGLang、TGI、Triton、NIM);Python Truss是请求路径中任意代码的兜底方案;Chains添加了类型化的步骤间传输,内置限流、连接管理、结构化错误传播和二进制IO——这些功能若基于原生Truss实现则需自行开发。Python Truss和Chains支持实时补丁迭代();所有方式均支持每副本自动扩缩容、缩容至零,以及环境/版本升级。
truss watch| 您需要… | 方式 | 参考文档 | 不适用场景 |
|---|---|---|---|
| 托管LLM,无需部署步骤 | 模型API | | 模型不在目录中;需要自定义硬件、依赖、稳定性等... |
| 在现成服务器上部署LLM(vLLM / SGLang / TGI / Triton / NIM) | 自定义Docker服务器 | | 服务器不存在,或您需要在请求路径中加入Python代码 |
| 通过Baseten引擎(TRT-LLM / BEI / BIS-LLM)部署LLM/嵌入模型;配置极简,无需Python | 仅引擎的 | | 架构未被引擎覆盖;需要自定义逻辑 |
| 请求路径中加入自定义Python代码(预处理/后处理、自定义架构、特殊IO) | Python类Truss( | | 引擎或现成服务器可满足需求——优先选择,部署速度更快 |
| 多步骤流水线,异构硬件/按步骤扩缩容(RAG、ASR→LLM→TTS、扇出、分片) | Chains | | 单阶段或同构场景——单个Truss更简单 |
独立于上述方式的运维层面:
- 迭代/补丁部署 →
model-dev-loop.md - 版本升级、环境管理、自动扩缩容 →
deployment-lifecycle.md - 逐步将流量切换到新部署(暂停/恢复/取消)→ 滚动部署:
deployment/rolling-deployments.mdx - 调用部署 → (自定义部署)或
inference-api.md(托管部署)model-apis.md - 高吞吐量批量调用(嵌入、重排序)→ 性能客户端(Rust,1200+请求/秒):
inference/performance-client.mdx - 程序化控制平面 →
management-api.md
表格未涵盖的实际场景细节:
- 混合方式存在。可封装引擎以实现预处理/后处理;Chain入口可以是Python类,而内部Chainlet使用引擎。
model.py - Chain WebSocket仅支持入口。Chainlet内部调用仅支持输出流,通常这些边缘不需要双向流。
- 引擎性能vs灵活性。TRT-LLM是许多LLM的最快路径,但其配置界面不够透明。仅当延迟/吞吐量是实际约束时,才值得权衡。
Routing
路由指引
Skill References ( in skill dir, complementary to hosted docs). Be generous to read any of the
included reference files as soon as the user touches on that topic.
ls references/- :
references/truss-cli.md/truss push/ iterate. Most-used. Deep dive:watch.references/truss-config.md - : Python-class flavor (custom pre/post, non-engine architectures).
references/truss-model-py.md - :
references/truss-custom-servers.mdflavor (vLLM / TGI / SGLang / Triton; most common modern-LLM path).docker_server - : multi-step pipelines (RAG, ASR→LLM→TTS, chunked audio/video) with per-step HW + autoscaling.
references/truss-chains.md - : shared pre-hosted endpoints (DeepSeek, GLM, Kimi, ...). Fastest when one fits.
references/model-apis.md - : calling custom deployments. connection reuse (
references/inference-api.md/requests/ OpenAI SDK), async / streaming / wake / OpenAI-compat sync routes.httpx - : programmatic control plane (models, deployments, envs, secrets). What
references/management-api.mdCLI uses under the hood.truss - : Model / Deployment / Environment semantics + promotion + autoscaling.
references/deployment-lifecycle.md - : post-first-deploy iteration. rebuild / patch / hot-reload cost tiers, agent-vs-human watch loop.
references/model-dev-loop.md
技能参考文档(技能目录下的,补充托管文档)。当用户涉及相关主题时,请主动阅读对应的参考文件。
ls references/- :
references/truss-cli.md/truss push/ 迭代。最常用。深入了解:watch。references/truss-config.md - :Python类方式(自定义预处理/后处理,非引擎架构)。
references/truss-model-py.md - :
references/truss-custom-servers.md方式(vLLM / TGI / SGLang / Triton;现代LLM最常用路径)。docker_server - :多步骤流水线(RAG、ASR→LLM→TTS、音频/视频分片),支持按步骤分配硬件+自动扩缩容。
references/truss-chains.md - :共享预托管端点(DeepSeek、GLM、Kimi等)。符合需求时是最快路径。
references/model-apis.md - :调用自定义部署。连接复用(
references/inference-api.md/requests/ OpenAI SDK)、异步/流式/唤醒/兼容OpenAI的同步路由。httpx - :程序化控制平面(模型、部署、环境、密钥)。
references/management-api.mdCLI底层依赖的接口。truss - :模型/部署/环境语义 + 版本升级 + 自动扩缩容。
references/deployment-lifecycle.md - :首次部署后的迭代。重建/补丁/热重载的成本层级,代理vs人工watch循环。
references/model-dev-loop.md
Gotchas
注意事项
Don't speculate, query
不要猜测,要查询
For any perf/status/error claim, use the tools first — don't estimate or guess.
- Timings → deployment / chainlet log tools (timestamped, includes ,
Pulling image,model_cache: Fetch took, per-request markers).Completed model.load() execution in N ms - Status → deployment-get tools before invoking.
- Build/deploy failure → fetch logs immediately, don't hypothesize.
Fabricating numbers from training-data priors burns user trust; logs are the source of truth.
对于任何性能/状态/错误问题,先使用工具——不要估算或猜测。
- 计时信息 → 部署/chainlet日志工具(带时间戳,包含、
Pulling image、model_cache: Fetch took、每个请求的标记)。Completed model.load() execution in N ms - 状态信息 → 调用前先使用部署查询工具。
- 构建/部署失败 → 立即获取日志,不要假设原因。
根据训练数据臆造数据会损害用户信任;日志是唯一的事实来源。
Source heterogeneity & drift
来源异构性与偏差
Content lives across systems that don't overlap cleanly and drift independently. No single source is
perfect/authoritative. For any non-trivial claim ("supported", perf numbers, recommended approach), triangulate across
≥2 sources. Surface contradictions to the user; don't paper over.
| Source | Strength | Gap / quirk |
|---|---|---|
| API specs, protocol details, knobs | Lags product; no perf numbers |
| Flagship managed models, perf claims | Some entries are sales-gated, not self-serve |
| What's actually one-click API-deployable | Doesn't include every marketing-library entry; lacking tags |
| Concrete latency / cost / vs-competitor numbers, technical deep dives | Unstructured; not in docs MCP. Discover via |
| High-level pitch | May describe flagship features that need a Baseten engagement |
| Working code patterns | Often outdated / broken / drifted. Consult with caution, last resort. Might need fixups before deploy works. |
- Library page exists but model absent from → likely managed/flagship; tell the user it may need to reach out to Baseten support.
list_library_models - Perf numbers found only in a blog → cite as blog claim.
- Can't find something via docs MCP → fetch or
baseten.co/llms.txtas index, then fetch the page directly. Last resort: web search.docs.baseten.co/llms.txt
内容分布在多个系统中,这些系统并非完全重叠,且各自独立更新。没有单一来源是完美/权威的。对于任何非琐碎的声明(“支持”、性能数据、推荐方案),至少从2个来源交叉验证。向用户指出矛盾之处;不要掩盖。
| 来源 | 优势 | 不足/特殊点 |
|---|---|---|
| API规范、协议细节、配置项 | 滞后于产品;无性能数据 |
| 旗舰托管模型、性能声明 | 部分内容需销售对接,无法自助使用 |
| 实际可一键API部署的模型 | 不包含所有营销库中的模型;缺少标签 |
| 具体延迟/成本/竞品对比数据、技术深度解析 | 非结构化;不在docs MCP中。可通过 |
| 高层级方案介绍 | 可能描述的是需要Baseten对接的旗舰功能 |
| 可用的代码模式 | 经常过时/损坏/偏离当前版本。谨慎参考,仅作为最后手段。部署前可能需要修复。 |
- 库页面存在但模型未在中 → 可能是托管/旗舰模型;告知用户可能需要联系Baseten支持。
list_library_models - 仅在博客中找到性能数据 → 注明是博客中的声明。
- 无法通过docs MCP找到内容 → 获取或
baseten.co/llms.txt作为索引,然后直接获取页面。最后手段:网页搜索。docs.baseten.co/llms.txt
Non-obvious placements within references/
references/references/
中不明显的内容位置
references/- Engine-only deploys (TensorRT-LLM, BEI, BIS-LLM) → engines section (also owns
truss-config.md, secrets, resources).model_cache - Authoring-flavor decision: single deployment → top of ; multiple coordinated →
truss-config.md.truss-chains.md - Training and Frontier Gateway: no skill reference. Use MCP +
basetenMCP. For training path choice seebaseten_docs; for Loops (managed SFT/RL SDK) seetraining/overview.mdx; for SSH / VS Code tunnels into training containers seeloops/overview.mdxandtraining/ssh.mdx.training/remote-access.mdx
- 仅引擎部署(TensorRT-LLM、BEI、BIS-LLM)→ 引擎章节(同时包含
truss-config.md、密钥、资源配置)。model_cache - 创作方式决策:单部署 → 顶部;多协同部署 →
truss-config.md。truss-chains.md - 训练和前沿网关:无技能参考文档。使用MCP +
basetenMCP。训练路径选择见baseten_docs;Loops(托管SFT/RL SDK)见training/overview.mdx;训练容器的SSH / VS Code隧道见loops/overview.mdx和training/ssh.mdx。training/remote-access.mdx
Tool quirks
工具特性
- MCP search is semantic (embedding-based). Results are generally intent-aware, but can still miss on niche phrasing — try rephrased queries or fetch
baseten_docsas an index if results are weak.docs.baseten.co/llms.txt - Full doc pages: /
catonheadpaths via.mdx. Alternatively,query_docs_filesystem_basetenalso works and is faster for single-page reads.docs.baseten.co/<path>.md - Inference SSH (terminal in running container, requires org enablement) → § Inference SSH.
model-dev-loop.md - is baseten curated catalog (~tens of pre-optimized hosted models, mostly popular open-source LLMs / embeddings). Models are good starter models to play, but not for custom authoring, specific performance needs, finetuning and private HF models etc. No useful tags (modality etc.) — filter by
list_library_models/display_namesubstring.hf_repo_id - Blog content is not in the docs MCP. Fetch and search for relevant posts.
baseten.co/llms.txt
- MCP搜索是语义化的(基于嵌入)。结果通常能匹配意图,但仍可能遗漏小众表述——若结果不佳,尝试重新表述查询或获取
baseten_docs作为索引。docs.baseten.co/llms.txt - 完整文档页面:通过对
query_docs_filesystem_baseten路径执行.mdx/cat操作。或者,head也可用,单页面读取速度更快。docs.baseten.co/<path>.md - 推理SSH(运行中容器的终端,需组织启用)→ 中的“推理SSH”章节。
model-dev-loop.md - 是Baseten curated目录(约数十个预优化的托管模型,多为热门开源LLM/嵌入模型)。这些模型适合入门试用,但不适用于自定义创作、特定性能需求、微调及私有HF模型等场景。无有用标签(如模态等)——可通过
list_library_models/display_name子串过滤。hf_repo_id - 博客内容不在docs MCP中。获取并搜索相关文章。
baseten.co/llms.txt
MCP introspection vs. invocation
MCP内省与调用
Backend MCP tools (, , , , …) describe state. They
are strongest when you need an exact id / schema / config value to act on, and weakest when the user's complaint is
about runtime behavior. Resist letting MCP availability shrink the size of your investigation.
get_deploymentget_deployment_logsget_deployment_configlist_*- Look-up tasks (need an ID, current config, or schema before writing a patch): MCP get/list is the right starting point. Reading the live config beats inferring it.
- Behavior tasks (broken / slow / wrong-shape responses, anything the user observed): MCP can supplement but
rarely substitutes for exercising the endpoint yourself. A against the predict URL with a representative payload runs the same code path the user does. Absence of recent errors in MCP-fetched logs is not proof the bug is gone — log windows are bounded, and a problem the user reported five minutes ago may have already aged out of the default tail. For a specific historic incident you can't reproduce, log tools support custom time ranges within retention; use them.
curl - Treat cheap introspection as budget freed up for deeper verification (re-curl after a fix, diff configs before and after a PATCH, push a candidate truss and probe it), not as license to stop earlier.
后端MCP工具(、、、等)描述状态。当您需要确切的ID/ schema/配置值来执行操作时,它们最有用;当用户反馈的是运行时行为问题时,它们作用最弱。不要因为MCP可用就缩小调查范围。
get_deploymentget_deployment_logsget_deployment_configlist_*- 查询任务(需要ID、当前配置或schema来编写补丁):MCP的get/list是正确起点。读取实时配置优于推断。
- 行为任务(响应异常/缓慢/格式错误,用户观察到的任何问题):MCP可作为补充,但很少能替代您自己调用端点测试。使用代表性payload对预测URL执行请求,走的是与用户相同的代码路径。MCP获取的日志中没有近期错误,不能证明问题已解决——日志窗口是有限的,用户5分钟前报告的问题可能已超出默认日志保留范围。对于无法重现的特定历史事件,日志工具支持在保留期内自定义时间范围;请使用该功能。
curl - 将低成本的内省视为可用于深度验证的预算(修复后重新curl,PATCH前后对比配置,推送候选truss并测试),而非提前结束调查的理由。