baseten

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Baseten Product Overview

Baseten产品概述

Production AI inference platform - serve and scale open-source, custom, and fine-tuned models with the fastest runtimes, cross-cloud HA, and seamless developer workflows.
  • Dedicated Inference - deploy any model, performance-optimized + horizontally scaled. Authored as auto-wrapped Truss server, custom Docker server, or compound/orchestrated deployment via Chains.
  • Model APIs - pre-optimized hosted APIs for popular models. Path to graduate to dedicated.
  • Training - two paths: Truss Train (BYO container, any framework, full hardware control) and Loops (Tinker-compatible managed SDK for SFT + async RL; paired trainer + sampling server, live weight transfers, one-click checkpoint deploy). Multi-node, 1T+ params, 10TB+ datasets, H100/H200/B200. Remote access: SSH and VS Code/Cursor tunnels into containers.
  • Frontier Gateway - operate your own foundation model B2C.
生产级AI推理平台——凭借最快的运行时、跨云高可用(HA)以及流畅的开发者工作流,部署并扩展开源、自定义和微调模型。
  • 专属推理 - 部署任意模型,性能优化+水平扩展。可通过自动封装的Truss服务器、自定义Docker服务器,或是借助Chains实现复合/编排式部署。
  • 模型API - 针对热门模型预优化的托管API。可升级至专属部署。
  • 训练 - 两种路径:Truss Train(自带容器,支持任意框架,完全硬件控制)和Loops(兼容Tinker的托管SDK,用于SFT+异步RL;配对训练器+采样服务器,实时权重传输,一键部署检查点)。支持多节点、1T+参数、10TB+数据集,适配H100/H200/B200硬件。远程访问:通过SSH和VS Code/Cursor隧道连接容器。
  • 前沿网关(Frontier Gateway) - 运营您自己的大模型B2C业务。

Agent DX Toolkit

Agent DX工具包

ComponentProvidesInstall
baseten
MCP
Interact with backend (~REST API, CRUD): models, deployments, training, environments, secrets, chains. API-key auth.
npx add-mcp https://api.baseten.co/mcp -g -y --header "Authorization: Bearer ${BASETEN_MCP_KEY}"
baseten_docs
MCP
Semantic search + filesystem of
docs.baseten.co
. No auth.
npx add-mcp https://docs.baseten.co/mcp -n "baseten_docs" -g -y
truss
CLI
Needed for model/chain push from local code, watch (= live patch). Needs
truss login
once.
uv tool install truss
(or
pip install truss --upgrade
; respect user package manager: uv, poetry...)
llms.txt
baseten.co/llms.txt
(product + blog),
docs.baseten.co/llms.txt
(docs).
reachable via HTTP
This skill
SKILL.md
+
references/*.md
loaded on demand.
npx skills add basetenlabs/baseten-skills -g -y
组件功能安装方式
baseten
MCP
与后端交互(类REST API,CRUD操作):模型、部署、训练、环境、密钥、Chains。API密钥认证。
npx add-mcp https://api.baseten.co/mcp -g -y --header "Authorization: Bearer ${BASETEN_MCP_KEY}"
baseten_docs
MCP
docs.baseten.co
的语义搜索+文件系统。无需认证。
npx add-mcp https://docs.baseten.co/mcp -n "baseten_docs" -g -y
truss
CLI
用于从本地代码推送模型/Chain、实时补丁(watch)。需执行一次
truss login
uv tool install truss
(或
pip install truss --upgrade
;尊重用户包管理器:uv、poetry等)
llms.txt
baseten.co/llms.txt
(产品+博客)、
docs.baseten.co/llms.txt
(文档)。
可通过HTTP访问
本技能按需加载
SKILL.md
+
references/*.md
npx skills add basetenlabs/baseten-skills -g -y

Setup

安装设置

  • Any subset works, full install recommended.
  • Suggest additional installs when the current task benefits from or requires them; help user with installation, but elicit preferences first.
  • Ensure
    BASETEN_MCP_KEY
    is provided when installing Baseten MCP (user can create key at
    app.baseten.co/settings/api_keys
    ). Caveat: an MCP instance binds to one org/workspace at install time; switching the bound workspace later is not supported. To work with multiple workspaces, install additional MCP instances under different names with different keys (see last bullet of this section).
  • Truss CLI only needed for making deployments (check
    truss --version
    ); prior login (multi-workspace users must provide
    --remote <name>
    ). Explore with
    truss [subcommand] --help
    .
  • Docs MCP missing → grep / fetch
    llms.txt
    .
  • Backend MCP is API-key-only (currently); OAuth-only harnesses can still use the other components.
  • If backend MCP server is needed for different orgs/workspaces, add multiple MCP instances with different names/keys or use env-var expansion in the agent's config file and set the env var to the respective workspace's key.
  • 可选择安装任意子集,推荐完整安装。
  • 当当前任务需要或受益于额外组件时,建议用户安装;协助用户安装,但需先询问用户偏好。
  • 安装Baseten MCP时需提供
    BASETEN_MCP_KEY
    (用户可在
    app.baseten.co/settings/api_keys
    创建密钥)。注意:MCP实例在安装时绑定到一个组织/工作区;后续不支持切换绑定的工作区。如需在多个工作区操作,请使用不同密钥安装多个同名不同配置的MCP实例(见本节最后一点)。
  • Truss CLI仅在创建部署时需要(检查
    truss --version
    );首次登录(多工作区用户需指定
    --remote <name>
    )。可通过
    truss [子命令] --help
    查看详情。
  • 若缺少Docs MCP → 搜索/获取
    llms.txt
  • 后端MCP目前仅支持API密钥认证;仅支持OAuth的工具仍可使用其他组件。
  • 若需为不同组织/工作区使用后端MCP服务器,可添加多个不同名称/密钥的MCP实例,或在代理配置文件中使用环境变量扩展,并将环境变量设置为对应工作区的密钥。

Pick your authoring surface (for creating deployments)

选择部署创作方式

First-pass decision. Many real workloads blend rows — treat this as a starting point, not a rule. When unsure, sketch the IO shape and per-step hardware needs before picking.
Surfaces are stacked by opinion-strength, not just author convenience: engines (TRT-LLM, BEI, BIS-LLM) ship performance-tuned for one architecture and are the fastest path when they fit; custom Docker servers wrap mature inference servers (vLLM, SGLang, TGI, Triton, NIM); Python Truss is the escape hatch for arbitrary code in the request path; Chains add typed inter-step transport with built-in rate limiting, connection management, structured error propagation, and binary IO — features you'd otherwise rebuild around N raw Trusses. Python Truss and Chains share live-patch iteration (
truss watch
); all flavors support per-replica autoscaling, scale-to-zero, and environments / promotions.
You want…FlavorSpecializationWhen NOT to pick
Hosted LLM, no deploy stepModel APIs
model-apis.md
model not in catalog; need custom hardware, requirements, stability...
LLM on an off-the-shelf server (vLLM / SGLang / TGI / Triton / NIM)Custom Docker server
truss-custom-servers.md
the server doesn't exist or you need Python in the request path
LLM/embedding via a Baseten engine (TRT-LLM / BEI / BIS-LLM); minimal config, no PythonEngine-only
config.yaml
truss-config.md
(engines section)
architecture not covered by an engine; you need custom logic
Custom Python in the request path (pre/post, custom arch, weird IO)Python-class Truss (
model.py
)
truss-model-py.md
+
truss-config.md
an engine or off-the-shelf server fits — pick that, it's faster to ship
Multi-step pipeline with heterogeneous hardware / per-step scaling (RAG, ASR→LLM→TTS, fan-out, chunking)Chains
truss-chains.md
one-stage or homogeneous — a single Truss is simpler
Orthogonal operational surfaces (independent of which flavor above):
  • Iterate / patch a deployment →
    model-dev-loop.md
  • Promote, environments, autoscaling →
    deployment-lifecycle.md
  • Gradual traffic shift to new deployment (pause/resume/cancel) → rolling deployments:
    deployment/rolling-deployments.mdx
  • Call a deployment →
    inference-api.md
    (custom) or
    model-apis.md
    (hosted)
  • High-throughput batch calls (embeddings, reranking) → Performance client (Rust, 1200+ req/s):
    inference/performance-client.mdx
  • Programmatic control plane →
    management-api.md
Real-world nuances the table can't capture:
  • Hybrids exist. A
    model.py
    can wrap an engine for pre/post-processing; a Chain entrypoint can be a Python class while internal Chainlets use engines.
  • Chain websockets are entrypoint-only. Intra-chainlet calls only stream output, but bi-di usually not needed on those edges.
  • Engine performance vs flexibility. TRT-LLM is the fastest path for many LLMs but its config surface is opaque. Worth the trade only when latency/throughput is a real constraint.
初步决策。许多实际工作负载会混合多种方式——将此视为起点,而非规则。不确定时,先勾勒IO形态和每一步的硬件需求再做选择。
各种方式按推荐优先级排序,而非仅考虑作者便利性:引擎(TRT-LLM、BEI、BIS-LLM)针对特定架构进行性能调优,适配时是最快路径;自定义Docker服务器封装成熟的推理服务器(vLLM、SGLang、TGI、Triton、NIM);Python Truss是请求路径中任意代码的兜底方案;Chains添加了类型化的步骤间传输,内置限流、连接管理、结构化错误传播和二进制IO——这些功能若基于原生Truss实现则需自行开发。Python Truss和Chains支持实时补丁迭代(
truss watch
);所有方式均支持每副本自动扩缩容、缩容至零,以及环境/版本升级。
您需要…方式参考文档不适用场景
托管LLM,无需部署步骤模型API
model-apis.md
模型不在目录中;需要自定义硬件、依赖、稳定性等...
在现成服务器上部署LLM(vLLM / SGLang / TGI / Triton / NIM)自定义Docker服务器
truss-custom-servers.md
服务器不存在,或您需要在请求路径中加入Python代码
通过Baseten引擎(TRT-LLM / BEI / BIS-LLM)部署LLM/嵌入模型;配置极简,无需Python仅引擎的
config.yaml
truss-config.md
(引擎章节)
架构未被引擎覆盖;需要自定义逻辑
请求路径中加入自定义Python代码(预处理/后处理、自定义架构、特殊IO)Python类Truss(
model.py
truss-model-py.md
+
truss-config.md
引擎或现成服务器可满足需求——优先选择,部署速度更快
多步骤流水线,异构硬件/按步骤扩缩容(RAG、ASR→LLM→TTS、扇出、分片)Chains
truss-chains.md
单阶段或同构场景——单个Truss更简单
独立于上述方式的运维层面:
  • 迭代/补丁部署 →
    model-dev-loop.md
  • 版本升级、环境管理、自动扩缩容 →
    deployment-lifecycle.md
  • 逐步将流量切换到新部署(暂停/恢复/取消)→ 滚动部署:
    deployment/rolling-deployments.mdx
  • 调用部署 →
    inference-api.md
    (自定义部署)或
    model-apis.md
    (托管部署)
  • 高吞吐量批量调用(嵌入、重排序)→ 性能客户端(Rust,1200+请求/秒):
    inference/performance-client.mdx
  • 程序化控制平面 →
    management-api.md
表格未涵盖的实际场景细节:
  • 混合方式存在
    model.py
    可封装引擎以实现预处理/后处理;Chain入口可以是Python类,而内部Chainlet使用引擎。
  • Chain WebSocket仅支持入口。Chainlet内部调用仅支持输出流,通常这些边缘不需要双向流。
  • 引擎性能vs灵活性。TRT-LLM是许多LLM的最快路径,但其配置界面不够透明。仅当延迟/吞吐量是实际约束时,才值得权衡。

Routing

路由指引

Skill References (
ls references/
in skill dir, complementary to hosted docs). Be generous to read any of the included reference files as soon as the user touches on that topic.
  • references/truss-cli.md
    :
    truss push
    /
    watch
    / iterate. Most-used. Deep dive:
    references/truss-config.md
    .
  • references/truss-model-py.md
    : Python-class flavor (custom pre/post, non-engine architectures).
  • references/truss-custom-servers.md
    :
    docker_server
    flavor (vLLM / TGI / SGLang / Triton; most common modern-LLM path).
  • references/truss-chains.md
    : multi-step pipelines (RAG, ASR→LLM→TTS, chunked audio/video) with per-step HW + autoscaling.
  • references/model-apis.md
    : shared pre-hosted endpoints (DeepSeek, GLM, Kimi, ...). Fastest when one fits.
  • references/inference-api.md
    : calling custom deployments. connection reuse (
    requests
    /
    httpx
    / OpenAI SDK), async / streaming / wake / OpenAI-compat sync routes.
  • references/management-api.md
    : programmatic control plane (models, deployments, envs, secrets). What
    truss
    CLI uses under the hood.
  • references/deployment-lifecycle.md
    : Model / Deployment / Environment semantics + promotion + autoscaling.
  • references/model-dev-loop.md
    : post-first-deploy iteration. rebuild / patch / hot-reload cost tiers, agent-vs-human watch loop.
技能参考文档(技能目录下的
ls references/
,补充托管文档)。当用户涉及相关主题时,请主动阅读对应的参考文件。
  • references/truss-cli.md
    truss push
    /
    watch
    / 迭代。最常用。深入了解:
    references/truss-config.md
  • references/truss-model-py.md
    :Python类方式(自定义预处理/后处理,非引擎架构)。
  • references/truss-custom-servers.md
    docker_server
    方式(vLLM / TGI / SGLang / Triton;现代LLM最常用路径)。
  • references/truss-chains.md
    :多步骤流水线(RAG、ASR→LLM→TTS、音频/视频分片),支持按步骤分配硬件+自动扩缩容。
  • references/model-apis.md
    :共享预托管端点(DeepSeek、GLM、Kimi等)。符合需求时是最快路径。
  • references/inference-api.md
    :调用自定义部署。连接复用(
    requests
    /
    httpx
    / OpenAI SDK)、异步/流式/唤醒/兼容OpenAI的同步路由。
  • references/management-api.md
    :程序化控制平面(模型、部署、环境、密钥)。
    truss
    CLI底层依赖的接口。
  • references/deployment-lifecycle.md
    :模型/部署/环境语义 + 版本升级 + 自动扩缩容。
  • references/model-dev-loop.md
    :首次部署后的迭代。重建/补丁/热重载的成本层级,代理vs人工watch循环。

Gotchas

注意事项

Don't speculate, query

不要猜测,要查询

For any perf/status/error claim, use the tools first — don't estimate or guess.
  • Timings → deployment / chainlet log tools (timestamped, includes
    Pulling image
    ,
    model_cache: Fetch took
    ,
    Completed model.load() execution in N ms
    , per-request markers).
  • Status → deployment-get tools before invoking.
  • Build/deploy failure → fetch logs immediately, don't hypothesize.
Fabricating numbers from training-data priors burns user trust; logs are the source of truth.
对于任何性能/状态/错误问题,先使用工具——不要估算或猜测。
  • 计时信息 → 部署/chainlet日志工具(带时间戳,包含
    Pulling image
    model_cache: Fetch took
    Completed model.load() execution in N ms
    、每个请求的标记)。
  • 状态信息 → 调用前先使用部署查询工具。
  • 构建/部署失败 → 立即获取日志,不要假设原因。
根据训练数据臆造数据会损害用户信任;日志是唯一的事实来源。

Source heterogeneity & drift

来源异构性与偏差

Content lives across systems that don't overlap cleanly and drift independently. No single source is perfect/authoritative. For any non-trivial claim ("supported", perf numbers, recommended approach), triangulate across ≥2 sources. Surface contradictions to the user; don't paper over.
SourceStrengthGap / quirk
baseten_docs
MCP /
docs.baseten.co
API specs, protocol details, knobsLags product; no perf numbers
baseten.co/library/<id>
(marketing)
Flagship managed models, perf claimsSome entries are sales-gated, not self-serve
baseten
MCP
list_library_models
What's actually one-click API-deployableDoesn't include every marketing-library entry; lacking tags
baseten.co/blog/
Concrete latency / cost / vs-competitor numbers, technical deep divesUnstructured; not in docs MCP. Discover via
baseten.co/llms.txt
baseten.co/solutions/...
High-level pitchMay describe flagship features that need a Baseten engagement
truss-examples
GitHub repo
Working code patternsOften outdated / broken / drifted. Consult with caution, last resort. Might need fixups before deploy works.
  • Library page exists but model absent from
    list_library_models
    → likely managed/flagship; tell the user it may need to reach out to Baseten support.
  • Perf numbers found only in a blog → cite as blog claim.
  • Can't find something via docs MCP → fetch
    baseten.co/llms.txt
    or
    docs.baseten.co/llms.txt
    as index, then fetch the page directly. Last resort: web search.
内容分布在多个系统中,这些系统并非完全重叠,且各自独立更新。没有单一来源是完美/权威的。对于任何非琐碎的声明(“支持”、性能数据、推荐方案),至少从2个来源交叉验证。向用户指出矛盾之处;不要掩盖。
来源优势不足/特殊点
baseten_docs
MCP /
docs.baseten.co
API规范、协议细节、配置项滞后于产品;无性能数据
baseten.co/library/<id>
(营销页面)
旗舰托管模型、性能声明部分内容需销售对接,无法自助使用
baseten
MCP
list_library_models
实际可一键API部署的模型不包含所有营销库中的模型;缺少标签
baseten.co/blog/
具体延迟/成本/竞品对比数据、技术深度解析非结构化;不在docs MCP中。可通过
baseten.co/llms.txt
发现
baseten.co/solutions/...
高层级方案介绍可能描述的是需要Baseten对接的旗舰功能
truss-examples
GitHub仓库
可用的代码模式经常过时/损坏/偏离当前版本。谨慎参考,仅作为最后手段。部署前可能需要修复。
  • 库页面存在但模型未在
    list_library_models
    中 → 可能是托管/旗舰模型;告知用户可能需要联系Baseten支持。
  • 仅在博客中找到性能数据 → 注明是博客中的声明。
  • 无法通过docs MCP找到内容 → 获取
    baseten.co/llms.txt
    docs.baseten.co/llms.txt
    作为索引,然后直接获取页面。最后手段:网页搜索。

Non-obvious placements within
references/

references/
中不明显的内容位置

  • Engine-only deploys (TensorRT-LLM, BEI, BIS-LLM) →
    truss-config.md
    engines section (also owns
    model_cache
    , secrets, resources).
  • Authoring-flavor decision: single deployment → top of
    truss-config.md
    ; multiple coordinated →
    truss-chains.md
    .
  • Training and Frontier Gateway: no skill reference. Use
    baseten
    MCP +
    baseten_docs
    MCP. For training path choice see
    training/overview.mdx
    ; for Loops (managed SFT/RL SDK) see
    loops/overview.mdx
    ; for SSH / VS Code tunnels into training containers see
    training/ssh.mdx
    and
    training/remote-access.mdx
    .
  • 仅引擎部署(TensorRT-LLM、BEI、BIS-LLM)→
    truss-config.md
    引擎章节(同时包含
    model_cache
    、密钥、资源配置)。
  • 创作方式决策:单部署 →
    truss-config.md
    顶部;多协同部署 →
    truss-chains.md
  • 训练和前沿网关:无技能参考文档。使用
    baseten
    MCP +
    baseten_docs
    MCP。训练路径选择见
    training/overview.mdx
    ;Loops(托管SFT/RL SDK)见
    loops/overview.mdx
    ;训练容器的SSH / VS Code隧道见
    training/ssh.mdx
    training/remote-access.mdx

Tool quirks

工具特性

  • baseten_docs
    MCP search is semantic (embedding-based). Results are generally intent-aware, but can still miss on niche phrasing — try rephrased queries or fetch
    docs.baseten.co/llms.txt
    as an index if results are weak.
  • Full doc pages:
    cat
    /
    head
    on
    .mdx
    paths via
    query_docs_filesystem_baseten
    . Alternatively,
    docs.baseten.co/<path>.md
    also works and is faster for single-page reads.
  • Inference SSH (terminal in running container, requires org enablement) →
    model-dev-loop.md
    § Inference SSH.
  • list_library_models
    is baseten curated catalog (~tens of pre-optimized hosted models, mostly popular open-source LLMs / embeddings). Models are good starter models to play, but not for custom authoring, specific performance needs, finetuning and private HF models etc. No useful tags (modality etc.) — filter by
    display_name
    /
    hf_repo_id
    substring.
  • Blog content is not in the docs MCP. Fetch
    baseten.co/llms.txt
    and search for relevant posts.
  • baseten_docs
    MCP搜索是语义化的(基于嵌入)。结果通常能匹配意图,但仍可能遗漏小众表述——若结果不佳,尝试重新表述查询或获取
    docs.baseten.co/llms.txt
    作为索引。
  • 完整文档页面:通过
    query_docs_filesystem_baseten
    .mdx
    路径执行
    cat
    /
    head
    操作。或者,
    docs.baseten.co/<path>.md
    也可用,单页面读取速度更快。
  • 推理SSH(运行中容器的终端,需组织启用)→
    model-dev-loop.md
    中的“推理SSH”章节。
  • list_library_models
    Baseten curated目录(约数十个预优化的托管模型,多为热门开源LLM/嵌入模型)。这些模型适合入门试用,但不适用于自定义创作、特定性能需求、微调及私有HF模型等场景。无有用标签(如模态等)——可通过
    display_name
    /
    hf_repo_id
    子串过滤。
  • 博客内容不在docs MCP中。获取
    baseten.co/llms.txt
    并搜索相关文章。

MCP introspection vs. invocation

MCP内省与调用

Backend MCP tools (
get_deployment
,
get_deployment_logs
,
get_deployment_config
,
list_*
, …) describe state. They are strongest when you need an exact id / schema / config value to act on, and weakest when the user's complaint is about runtime behavior. Resist letting MCP availability shrink the size of your investigation.
  • Look-up tasks (need an ID, current config, or schema before writing a patch): MCP get/list is the right starting point. Reading the live config beats inferring it.
  • Behavior tasks (broken / slow / wrong-shape responses, anything the user observed): MCP can supplement but rarely substitutes for exercising the endpoint yourself. A
    curl
    against the predict URL with a representative payload runs the same code path the user does. Absence of recent errors in MCP-fetched logs is not proof the bug is gone — log windows are bounded, and a problem the user reported five minutes ago may have already aged out of the default tail. For a specific historic incident you can't reproduce, log tools support custom time ranges within retention; use them.
  • Treat cheap introspection as budget freed up for deeper verification (re-curl after a fix, diff configs before and after a PATCH, push a candidate truss and probe it), not as license to stop earlier.
后端MCP工具(
get_deployment
get_deployment_logs
get_deployment_config
list_*
等)描述状态。当您需要确切的ID/ schema/配置值来执行操作时,它们最有用;当用户反馈的是运行时行为问题时,它们作用最弱。不要因为MCP可用就缩小调查范围。
  • 查询任务(需要ID、当前配置或schema来编写补丁):MCP的get/list是正确起点。读取实时配置优于推断。
  • 行为任务(响应异常/缓慢/格式错误,用户观察到的任何问题):MCP可作为补充,但很少能替代您自己调用端点测试。使用代表性payload对预测URL执行
    curl
    请求,走的是与用户相同的代码路径。MCP获取的日志中没有近期错误,不能证明问题已解决——日志窗口是有限的,用户5分钟前报告的问题可能已超出默认日志保留范围。对于无法重现的特定历史事件,日志工具支持在保留期内自定义时间范围;请使用该功能。
  • 将低成本的内省视为可用于深度验证的预算(修复后重新curl,PATCH前后对比配置,推送候选truss并测试),而非提前结束调查的理由。