baseten

Baseten Product Overview

Production AI inference platform - serve and scale open-source, custom, and fine-tuned models with the fastest runtimes, cross-cloud HA, and seamless developer workflows.

Dedicated Inference - deploy any model, performance-optimized + horizontally scaled. Authored as auto-wrapped Truss server, custom Docker server, or compound/orchestrated deployment via Chains.
Model APIs - pre-optimized hosted APIs for popular models. Path to graduate to dedicated.
Training - two paths: Truss Train (BYO container, any framework, full hardware control) and Loops (Tinker-compatible managed SDK for SFT + async RL; paired trainer + sampling server, live weight transfers, one-click checkpoint deploy). Multi-node, 1T+ params, 10TB+ datasets, H100/H200/B200. Remote access: SSH and VS Code/Cursor tunnels into containers.
Frontier Gateway - operate your own foundation model B2C.

Agent DX Toolkit

Component	Provides	Install
`baseten` MCP	Interact with backend (~REST API, CRUD): models, deployments, training, environments, secrets, chains. API-key auth.	`npx add-mcp https://api.baseten.co/mcp -g -y --header "Authorization: Bearer ${BASETEN_MCP_KEY}"`
`baseten_docs` MCP	Semantic search + filesystem of `docs.baseten.co` . No auth.	`npx add-mcp https://docs.baseten.co/mcp -n "baseten_docs" -g -y`
`truss` CLI	Needed for model/chain push from local code, watch (= live patch). Needs `truss login` once.	`uv tool install truss` (or `pip install truss --upgrade` ; respect user package manager: uv, poetry...)
`llms.txt`	`baseten.co/llms.txt` (product + blog), `docs.baseten.co/llms.txt` (docs).	reachable via HTTP
This skill	`SKILL.md` + `references/*.md` loaded on demand.	`npx skills add basetenlabs/baseten-skills -g -y`

Setup

Any subset works, full install recommended.
Suggest additional installs when the current task benefits from or requires them; help user with installation, but elicit preferences first.
Ensure
```
BASETEN_MCP_KEY
```
is provided when installing Baseten MCP (user can create key at
```
app.baseten.co/settings/api_keys
```
). Caveat: an MCP instance binds to one org/workspace at install time; switching the bound workspace later is not supported. To work with multiple workspaces, install additional MCP instances under different names with different keys (see last bullet of this section).
Truss CLI only needed for making deployments (check
```
truss --version
```
); prior login (multi-workspace users must provide
```
--remote <name>
```
). Explore with
```
truss [subcommand] --help
```
.
Docs MCP missing → grep / fetch
```
llms.txt
```
.
Backend MCP is API-key-only (currently); OAuth-only harnesses can still use the other components.
If backend MCP server is needed for different orgs/workspaces, add multiple MCP instances with different names/keys or use env-var expansion in the agent's config file and set the env var to the respective workspace's key.

Pick your authoring surface (for creating deployments)

First-pass decision. Many real workloads blend rows — treat this as a starting point, not a rule. When unsure, sketch the IO shape and per-step hardware needs before picking.

Surfaces are stacked by opinion-strength, not just author convenience: engines (TRT-LLM, BEI, BIS-LLM) ship performance-tuned for one architecture and are the fastest path when they fit; custom Docker servers wrap mature inference servers (vLLM, SGLang, TGI, Triton, NIM); Python Truss is the escape hatch for arbitrary code in the request path; Chains add typed inter-step transport with built-in rate limiting, connection management, structured error propagation, and binary IO — features you'd otherwise rebuild around N raw Trusses. Python Truss and Chains share live-patch iteration (

truss watch

); all flavors support per-replica autoscaling, scale-to-zero, and environments / promotions.

You want…	Flavor	Specialization	When NOT to pick
Hosted LLM, no deploy step	Model APIs	`model-apis.md`	model not in catalog; need custom hardware, requirements, stability...
LLM on an off-the-shelf server (vLLM / SGLang / TGI / Triton / NIM)	Custom Docker server	`truss-custom-servers.md`	the server doesn't exist or you need Python in the request path
LLM/embedding via a Baseten engine (TRT-LLM / BEI / BIS-LLM); minimal config, no Python	Engine-only `config.yaml`	`truss-config.md` (engines section)	architecture not covered by an engine; you need custom logic
Custom Python in the request path (pre/post, custom arch, weird IO)	Python-class Truss ( `model.py` )	`truss-model-py.md` + `truss-config.md`	an engine or off-the-shelf server fits — pick that, it's faster to ship
Multi-step pipeline with heterogeneous hardware / per-step scaling (RAG, ASR→LLM→TTS, fan-out, chunking)	Chains	`truss-chains.md`	one-stage or homogeneous — a single Truss is simpler

Orthogonal operational surfaces (independent of which flavor above):

Iterate / patch a deployment →
```
model-dev-loop.md
```
Promote, environments, autoscaling →
```
deployment-lifecycle.md
```
Gradual traffic shift to new deployment (pause/resume/cancel) → rolling deployments:
```
deployment/rolling-deployments.mdx
```
Call a deployment →
```
inference-api.md
```
(custom) or
```
model-apis.md
```
(hosted)
High-throughput batch calls (embeddings, reranking) → Performance client (Rust, 1200+ req/s):
```
inference/performance-client.mdx
```
Programmatic control plane →
```
management-api.md
```

Real-world nuances the table can't capture:

Hybrids exist. A
```
model.py
```
can wrap an engine for pre/post-processing; a Chain entrypoint can be a Python class while internal Chainlets use engines.
Chain websockets are entrypoint-only. Intra-chainlet calls only stream output, but bi-di usually not needed on those edges.
Engine performance vs flexibility. TRT-LLM is the fastest path for many LLMs but its config surface is opaque. Worth the trade only when latency/throughput is a real constraint.

Routing

Skill References (

ls references/

in skill dir, complementary to hosted docs). Be generous to read any of the included reference files as soon as the user touches on that topic.

references/truss-cli.md

truss push

watch

/ iterate. Most-used. Deep dive:

references/truss-config.md

```
references/truss-model-py.md
```
: Python-class flavor (custom pre/post, non-engine architectures).
```
references/truss-custom-servers.md
```
:
```
docker_server
```
flavor (vLLM / TGI / SGLang / Triton; most common modern-LLM path).
```
references/truss-chains.md
```
: multi-step pipelines (RAG, ASR→LLM→TTS, chunked audio/video) with per-step HW + autoscaling.
```
references/model-apis.md
```
: shared pre-hosted endpoints (DeepSeek, GLM, Kimi, ...). Fastest when one fits.
```
references/inference-api.md
```
: calling custom deployments. connection reuse (
```
requests
```
/
```
httpx
```
/ OpenAI SDK), async / streaming / wake / OpenAI-compat sync routes.
```
references/management-api.md
```
: programmatic control plane (models, deployments, envs, secrets). What
```
truss
```
CLI uses under the hood.
```
references/deployment-lifecycle.md
```
: Model / Deployment / Environment semantics + promotion + autoscaling.
```
references/model-dev-loop.md
```
: post-first-deploy iteration. rebuild / patch / hot-reload cost tiers, agent-vs-human watch loop.

Gotchas

Don't speculate, query

For any perf/status/error claim, use the tools first — don't estimate or guess.

Timings → deployment / chainlet log tools (timestamped, includes
```
Pulling image
```
,
```
model_cache: Fetch took
```
,
```
Completed model.load() execution in N ms
```
, per-request markers).
Status → deployment-get tools before invoking.
Build/deploy failure → fetch logs immediately, don't hypothesize.

Fabricating numbers from training-data priors burns user trust; logs are the source of truth.

Source heterogeneity & drift

Content lives across systems that don't overlap cleanly and drift independently. No single source is perfect/authoritative. For any non-trivial claim ("supported", perf numbers, recommended approach), triangulate across ≥2 sources. Surface contradictions to the user; don't paper over.

Source	Strength	Gap / quirk
`baseten_docs` MCP / `docs.baseten.co`	API specs, protocol details, knobs	Lags product; no perf numbers
`baseten.co/library/<id>` (marketing)	Flagship managed models, perf claims	Some entries are sales-gated, not self-serve
`baseten` MCP `list_library_models`	What's actually one-click API-deployable	Doesn't include every marketing-library entry; lacking tags
`baseten.co/blog/`	Concrete latency / cost / vs-competitor numbers, technical deep dives	Unstructured; not in docs MCP. Discover via `baseten.co/llms.txt`
`baseten.co/solutions/...`	High-level pitch	May describe flagship features that need a Baseten engagement
`truss-examples` GitHub repo	Working code patterns	Often outdated / broken / drifted. Consult with caution, last resort. Might need fixups before deploy works.

Library page exists but model absent from
```
list_library_models
```
→ likely managed/flagship; tell the user it may need to reach out to Baseten support.
Perf numbers found only in a blog → cite as blog claim.
Can't find something via docs MCP → fetch
```
baseten.co/llms.txt
```
or
```
docs.baseten.co/llms.txt
```
as index, then fetch the page directly. Last resort: web search.

Non-obvious placements within

references/

Engine-only deploys (TensorRT-LLM, BEI, BIS-LLM) →
```
truss-config.md
```
engines section (also owns
```
model_cache
```
, secrets, resources).
Authoring-flavor decision: single deployment → top of
```
truss-config.md
```
; multiple coordinated →
```
truss-chains.md
```
.
Training and Frontier Gateway: no skill reference. Use
```
baseten
```
MCP +
```
baseten_docs
```
MCP. For training path choice see
```
training/overview.mdx
```
; for Loops (managed SFT/RL SDK) see
```
loops/overview.mdx
```
; for SSH / VS Code tunnels into training containers see
```
training/ssh.mdx
```
and
```
training/remote-access.mdx
```
.

Tool quirks

```
baseten_docs
```
MCP search is semantic (embedding-based). Results are generally intent-aware, but can still miss on niche phrasing — try rephrased queries or fetch
```
docs.baseten.co/llms.txt
```
as an index if results are weak.
Full doc pages:
```
cat
```
/
```
head
```
on
```
.mdx
```
paths via
```
query_docs_filesystem_baseten
```
. Alternatively,
```
docs.baseten.co/<path>.md
```
also works and is faster for single-page reads.
Inference SSH (terminal in running container, requires org enablement) →
```
model-dev-loop.md
```
§ Inference SSH.
```
list_library_models
```
is baseten curated catalog (~tens of pre-optimized hosted models, mostly popular open-source LLMs / embeddings). Models are good starter models to play, but not for custom authoring, specific performance needs, finetuning and private HF models etc. No useful tags (modality etc.) — filter by
```
display_name
```
/
```
hf_repo_id
```
substring.
Blog content is not in the docs MCP. Fetch
```
baseten.co/llms.txt
```
and search for relevant posts.

MCP introspection vs. invocation

Backend MCP tools (

get_deployment

get_deployment_logs

get_deployment_config

list_*

, …) describe state. They are strongest when you need an exact id / schema / config value to act on, and weakest when the user's complaint is about runtime behavior. Resist letting MCP availability shrink the size of your investigation.

Look-up tasks (need an ID, current config, or schema before writing a patch): MCP get/list is the right starting point. Reading the live config beats inferring it.
Behavior tasks (broken / slow / wrong-shape responses, anything the user observed): MCP can supplement but rarely substitutes for exercising the endpoint yourself. A
```
curl
```
against the predict URL with a representative payload runs the same code path the user does. Absence of recent errors in MCP-fetched logs is not proof the bug is gone — log windows are bounded, and a problem the user reported five minutes ago may have already aged out of the default tail. For a specific historic incident you can't reproduce, log tools support custom time ranges within retention; use them.
Treat cheap introspection as budget freed up for deeper verification (re-curl after a fix, diff configs before and after a PATCH, push a candidate truss and probe it), not as license to stop earlier.