Multi-Tenant LLM Hosting

多租户LLM托管

Host many teams/customers on shared inference infrastructure without sacrificing security, performance, or cost governance.

在共享推理基础设施上为多个团队/客户提供服务，同时不牺牲安全性、性能或成本管控能力。

When to Use This Skill

何时使用该技能

Building an internal LLM platform shared by multiple teams
Hosting LLM inference for external customers with isolation requirements
Implementing per-tenant quotas, billing, and rate limiting
Designing request routing for multi-model, multi-tenant environments
Preventing noisy-neighbor issues on shared GPU infrastructure

构建供多个团队共享的内部LLM平台
为有隔离需求的外部客户提供LLM推理托管服务
实现租户级配额、账单和速率限制
为多模型、多租户环境设计请求路由机制
防止共享GPU基础设施上的噪声邻居问题

Prerequisites

前置条件

Kubernetes cluster with GPU node pools
API gateway or LLM gateway (LiteLLM, Envoy, Kong)
Prometheus + Grafana for per-tenant observability
Redis or equivalent for rate limiting state
Billing system or cost attribution database

带有GPU节点池的Kubernetes集群
API网关或LLM网关（LiteLLM、Envoy、Kong）
用于租户级可观测性的Prometheus + Grafana
用于速率限制状态存储的Redis或同类工具
账单系统或成本归因数据库

Isolation Model

隔离模型

Strong tenant identity on every request
Per-tenant API keys and scoped model access
Namespace or workload isolation for high-risk tenants
Strict data retention and log partitioning controls

每个请求都带有强租户身份标识
租户级API密钥和限定范围的模型访问权限
针对高风险租户的命名空间或工作负载隔离
严格的数据保留和日志分区控制

vLLM Multi-Model Serving

vLLM多模型服务

yaml

undefined

yaml

undefined

vllm-deployment.yaml - Multi-model serving with vLLM

apiVersion: apps/v1 kind: Deployment metadata: name: vllm-gpt4o-equivalent namespace: llm-serving labels: app: vllm model-tier: premium spec: replicas: 3 selector: matchLabels: app: vllm model-tier: premium template: metadata: labels: app: vllm model-tier: premium annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" spec: containers: - name: vllm image: vllm/vllm-openai:v0.4.1 args: - "--model=/models/llama-3.1-70b" - "--tensor-parallel-size=2" - "--max-model-len=8192" - "--gpu-memory-utilization=0.90" - "--max-num-seqs=128" - "--enable-prefix-caching" ports: - containerPort: 8000 name: inference - containerPort: 8080 name: metrics resources: requests: nvidia.com/gpu: 2 cpu: "8" memory: "64Gi" limits: nvidia.com/gpu: 2 cpu: "16" memory: "128Gi" volumeMounts: - name: model-weights mountPath: /models readOnly: true volumes: - name: model-weights persistentVolumeClaim: claimName: premium-model-weights tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule nodeSelector: gpu-type: a100

apiVersion: apps/v1 kind: Deployment metadata: name: vllm-economy namespace: llm-serving labels: app: vllm model-tier: economy spec: replicas: 2 selector: matchLabels: app: vllm model-tier: economy template: metadata: labels: app: vllm model-tier: economy spec: containers: - name: vllm image: vllm/vllm-openai:v0.4.1 args: - "--model=/models/llama-3.1-8b" - "--max-model-len=4096" - "--gpu-memory-utilization=0.85" - "--max-num-seqs=256" - "--enable-prefix-caching" ports: - containerPort: 8000 name: inference - containerPort: 8080 name: metrics resources: requests: nvidia.com/gpu: 1 cpu: "4" memory: "32Gi" limits: nvidia.com/gpu: 1 cpu: "8" memory: "64Gi" volumeMounts: - name: model-weights mountPath: /models readOnly: true volumes: - name: model-weights persistentVolumeClaim: claimName: economy-model-weights tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule

undefined

apiVersion: apps/v1 kind: Deployment metadata: name: vllm-gpt4o-equivalent namespace: llm-serving labels: app: vllm model-tier: premium spec: replicas: 3 selector: matchLabels: app: vllm model-tier: premium template: metadata: labels: app: vllm model-tier: premium annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" spec: containers: - name: vllm image: vllm/vllm-openai:v0.4.1 args: - "--model=/models/llama-3.1-70b" - "--tensor-parallel-size=2" - "--max-model-len=8192" - "--gpu-memory-utilization=0.90" - "--max-num-seqs=128" - "--enable-prefix-caching" ports: - containerPort: 8000 name: inference - containerPort: 8080 name: metrics resources: requests: nvidia.com/gpu: 2 cpu: "8" memory: "64Gi" limits: nvidia.com/gpu: 2 cpu: "16" memory: "128Gi" volumeMounts: - name: model-weights mountPath: /models readOnly: true volumes: - name: model-weights persistentVolumeClaim: claimName: premium-model-weights tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule nodeSelector: gpu-type: a100

apiVersion: apps/v1 kind: Deployment metadata: name: vllm-economy namespace: llm-serving labels: app: vllm model-tier: economy spec: replicas: 2 selector: matchLabels: app: vllm model-tier: economy template: metadata: labels: app: vllm model-tier: economy spec: containers: - name: vllm image: vllm/vllm-openai:v0.4.1 args: - "--model=/models/llama-3.1-8b" - "--max-model-len=4096" - "--gpu-memory-utilization=0.85" - "--max-num-seqs=256" - "--enable-prefix-caching" ports: - containerPort: 8000 name: inference - containerPort: 8080 name: metrics resources: requests: nvidia.com/gpu: 1 cpu: "4" memory: "32Gi" limits: nvidia.com/gpu: 1 cpu: "8" memory: "64Gi" volumeMounts: - name: model-weights mountPath: /models readOnly: true volumes: - name: model-weights persistentVolumeClaim: claimName: economy-model-weights tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule

undefined

Per-Tenant Quota Configuration

租户级配额配置

yaml

undefined

yaml

undefined

tenant-quotas-configmap.yaml

apiVersion: v1 kind: ConfigMap metadata: name: tenant-quotas namespace: llm-serving data: quotas.yaml: | tenants: acme-corp: tier: enterprise models_allowed: - llama-3.1-70b - llama-3.1-8b - nomic-embed-text rate_limits: requests_per_minute: 300 tokens_per_minute: 500000 concurrent_requests: 50 budget: daily_limit_usd: 500.00 monthly_limit_usd: 10000.00 alert_threshold_percent: 80 priority: high

  startup-xyz:
    tier: standard
    models_allowed:
      - llama-3.1-8b
      - nomic-embed-text
    rate_limits:
      requests_per_minute: 60
      tokens_per_minute: 100000
      concurrent_requests: 10
    budget:
      daily_limit_usd: 50.00
      monthly_limit_usd: 1000.00
      alert_threshold_percent: 80
    priority: medium

  internal-dev:
    tier: free
    models_allowed:
      - llama-3.1-8b
    rate_limits:
      requests_per_minute: 20
      tokens_per_minute: 50000
      concurrent_requests: 5
    budget:
      daily_limit_usd: 10.00
      monthly_limit_usd: 200.00
      alert_threshold_percent: 90
    priority: low

undefined

apiVersion: v1 kind: ConfigMap metadata: name: tenant-quotas namespace: llm-serving data: quotas.yaml: | tenants: acme-corp: tier: enterprise models_allowed: - llama-3.1-70b - llama-3.1-8b - nomic-embed-text rate_limits: requests_per_minute: 300 tokens_per_minute: 500000 concurrent_requests: 50 budget: daily_limit_usd: 500.00 monthly_limit_usd: 10000.00 alert_threshold_percent: 80 priority: high

  startup-xyz:
    tier: standard
    models_allowed:
      - llama-3.1-8b
      - nomic-embed-text
    rate_limits:
      requests_per_minute: 60
      tokens_per_minute: 100000
      concurrent_requests: 10
    budget:
      daily_limit_usd: 50.00
      monthly_limit_usd: 1000.00
      alert_threshold_percent: 80
    priority: medium

  internal-dev:
    tier: free
    models_allowed:
      - llama-3.1-8b
    rate_limits:
      requests_per_minute: 20
      tokens_per_minute: 50000
      concurrent_requests: 5
    budget:
      daily_limit_usd: 10.00
      monthly_limit_usd: 200.00
      alert_threshold_percent: 90
    priority: low

undefined

Namespace Isolation for High-Risk Tenants

高风险租户的命名空间隔离

yaml

undefined

yaml

undefined

tenant-namespace.yaml

apiVersion: v1 kind: Namespace metadata: name: tenant-acme-corp labels: tenant: acme-corp isolation: strict

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: tenant-isolation namespace: tenant-acme-corp spec: podSelector: {} policyTypes: - Ingress - Egress ingress: - from: - namespaceSelector: matchLabels: name: llm-gateway egress: - to: - namespaceSelector: matchLabels: name: llm-serving ports: - port: 8000 protocol: TCP - to: - namespaceSelector: matchLabels: name: kube-dns ports: - port: 53 protocol: UDP

apiVersion: v1 kind: ResourceQuota metadata: name: tenant-quota namespace: tenant-acme-corp spec: hard: requests.cpu: "16" requests.memory: "64Gi" limits.cpu: "32" limits.memory: "128Gi" requests.nvidia.com/gpu: "4" pods: "20"

undefined

apiVersion: v1 kind: Namespace metadata: name: tenant-acme-corp labels: tenant: acme-corp isolation: strict

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: tenant-isolation namespace: tenant-acme-corp spec: podSelector: {} policyTypes: - Ingress - Egress ingress: - from: - namespaceSelector: matchLabels: name: llm-gateway egress: - to: - namespaceSelector: matchLabels: name: llm-serving ports: - port: 8000 protocol: TCP - to: - namespaceSelector: matchLabels: name: kube-dns ports: - port: 53 protocol: UDP

apiVersion: v1 kind: ResourceQuota metadata: name: tenant-quota namespace: tenant-acme-corp spec: hard: requests.cpu: "16" requests.memory: "64Gi" limits.cpu: "32" limits.memory: "128Gi" requests.nvidia.com/gpu: "4" pods: "20"

undefined

Request Routing and Rate Limiting

请求路由与速率限制

python

undefined

python

undefined

gateway_router.py

"""Multi-tenant request router with rate limiting and model routing.""" import time import json import redis from fastapi import FastAPI, HTTPException, Header, Request from typing import Optional import httpx import yaml

app = FastAPI() redis_client = redis.Redis(host="redis", port=6379, decode_responses=True)

"""Multi-tenant request router with rate limiting and model routing.""" import time import json import redis from fastapi import FastAPI, HTTPException, Header, Request from typing import Optional import httpx import yaml

app = FastAPI() redis_client = redis.Redis(host="redis", port=6379, decode_responses=True)

Load tenant config

with open("/etc/config/quotas.yaml") as f: TENANT_CONFIG = yaml.safe_load(f)["tenants"]

MODEL_ENDPOINTS = { "llama-3.1-70b": "http://vllm-gpt4o-equivalent:8000", "llama-3.1-8b": "http://vllm-economy:8000", "nomic-embed-text": "http://embedding-service:8000", }

def check_rate_limit(tenant_id: str, config: dict) -> bool: """Check and update rate limit for a tenant.""" key = f"ratelimit:{tenant_id}:{int(time.time() // 60)}" current = redis_client.incr(key) if current == 1: redis_client.expire(key, 120) return current <= config["rate_limits"]["requests_per_minute"]

def check_concurrent(tenant_id: str, config: dict) -> bool: """Check concurrent request limit.""" key = f"concurrent:{tenant_id}" current = int(redis_client.get(key) or 0) return current < config["rate_limits"]["concurrent_requests"]

def check_budget(tenant_id: str, config: dict) -> bool: """Check if tenant is within daily budget.""" key = f"spend:{tenant_id}:{time.strftime('%Y-%m-%d')}" current_spend = float(redis_client.get(key) or 0) return current_spend < config["budget"]["daily_limit_usd"]

def record_usage(tenant_id: str, model: str, prompt_tokens: int, completion_tokens: int): """Record token usage and cost for billing.""" # Cost rates per 1K tokens rates = { "llama-3.1-70b": {"prompt": 0.004, "completion": 0.012}, "llama-3.1-8b": {"prompt": 0.0005, "completion": 0.0015}, "nomic-embed-text": {"prompt": 0.0001, "completion": 0.0}, } rate = rates.get(model, {"prompt": 0.001, "completion": 0.003}) cost = (prompt_tokens * rate["prompt"] + completion_tokens * rate["completion"]) / 1000

# Update daily spend
spend_key = f"spend:{tenant_id}:{time.strftime('%Y-%m-%d')}"
redis_client.incrbyfloat(spend_key, cost)
redis_client.expire(spend_key, 172800)

# Record for billing export
billing_key = f"billing:{tenant_id}:{time.strftime('%Y-%m')}"
redis_client.rpush(billing_key, json.dumps({
    "timestamp": time.time(),
    "model": model,
    "prompt_tokens": prompt_tokens,
    "completion_tokens": completion_tokens,
    "cost_usd": cost,
}))

@app.post("/v1/chat/completions") async def chat_completions( request: Request, x_tenant_id: str = Header(...), x_api_key: str = Header(...), ): """Route chat completion request with tenant controls.""" if x_tenant_id not in TENANT_CONFIG: raise HTTPException(status_code=403, detail="Unknown tenant")

config = TENANT_CONFIG[x_tenant_id]
body = await request.json()
model = body.get("model", "llama-3.1-8b")

# Check model access
if model not in config["models_allowed"]:
    raise HTTPException(status_code=403, detail=f"Model {model} not allowed for tenant")

# Check rate limit
if not check_rate_limit(x_tenant_id, config):
    raise HTTPException(status_code=429, detail="Rate limit exceeded")

# Check concurrent requests
if not check_concurrent(x_tenant_id, config):
    raise HTTPException(status_code=429, detail="Concurrent request limit exceeded")

# Check budget
if not check_budget(x_tenant_id, config):
    raise HTTPException(status_code=402, detail="Daily budget exceeded")

# Route to model endpoint
endpoint = MODEL_ENDPOINTS.get(model)
if not endpoint:
    raise HTTPException(status_code=404, detail=f"Model {model} not available")

# Track concurrent requests
concurrent_key = f"concurrent:{x_tenant_id}"
redis_client.incr(concurrent_key)

try:
    async with httpx.AsyncClient(timeout=120.0) as client:
        response = await client.post(
            f"{endpoint}/v1/chat/completions",
            json=body,
            headers={"Content-Type": "application/json"},
        )
        result = response.json()

        # Record usage
        usage = result.get("usage", {})
        record_usage(
            x_tenant_id, model,
            usage.get("prompt_tokens", 0),
            usage.get("completion_tokens", 0),
        )

        return result
finally:
    redis_client.decr(concurrent_key)

undefined

with open("/etc/config/quotas.yaml") as f: TENANT_CONFIG = yaml.safe_load(f)["tenants"]

MODEL_ENDPOINTS = { "llama-3.1-70b": "http://vllm-gpt4o-equivalent:8000", "llama-3.1-8b": "http://vllm-economy:8000", "nomic-embed-text": "http://embedding-service:8000", }

def check_rate_limit(tenant_id: str, config: dict) -> bool: """Check and update rate limit for a tenant.""" key = f"ratelimit:{tenant_id}:{int(time.time() // 60)}" current = redis_client.incr(key) if current == 1: redis_client.expire(key, 120) return current <= config["rate_limits"]["requests_per_minute"]

def check_concurrent(tenant_id: str, config: dict) -> bool: """Check concurrent request limit.""" key = f"concurrent:{tenant_id}" current = int(redis_client.get(key) or 0) return current < config["rate_limits"]["concurrent_requests"]

def check_budget(tenant_id: str, config: dict) -> bool: """Check if tenant is within daily budget.""" key = f"spend:{tenant_id}:{time.strftime('%Y-%m-%d')}" current_spend = float(redis_client.get(key) or 0) return current_spend < config["budget"]["daily_limit_usd"]

def record_usage(tenant_id: str, model: str, prompt_tokens: int, completion_tokens: int): """Record token usage and cost for billing.""" # Cost rates per 1K tokens rates = { "llama-3.1-70b": {"prompt": 0.004, "completion": 0.012}, "llama-3.1-8b": {"prompt": 0.0005, "completion": 0.0015}, "nomic-embed-text": {"prompt": 0.0001, "completion": 0.0}, } rate = rates.get(model, {"prompt": 0.001, "completion": 0.003}) cost = (prompt_tokens * rate["prompt"] + completion_tokens * rate["completion"]) / 1000

# Update daily spend
spend_key = f"spend:{tenant_id}:{time.strftime('%Y-%m-%d')}"
redis_client.incrbyfloat(spend_key, cost)
redis_client.expire(spend_key, 172800)

# Record for billing export
billing_key = f"billing:{tenant_id}:{time.strftime('%Y-%m')}"
redis_client.rpush(billing_key, json.dumps({
    "timestamp": time.time(),
    "model": model,
    "prompt_tokens": prompt_tokens,
    "completion_tokens": completion_tokens,
    "cost_usd": cost,
}))

@app.post("/v1/chat/completions") async def chat_completions( request: Request, x_tenant_id: str = Header(...), x_api_key: str = Header(...), ): """Route chat completion request with tenant controls.""" if x_tenant_id not in TENANT_CONFIG: raise HTTPException(status_code=403, detail="Unknown tenant")

config = TENANT_CONFIG[x_tenant_id]
body = await request.json()
model = body.get("model", "llama-3.1-8b")

# Check model access
if model not in config["models_allowed"]:
    raise HTTPException(status_code=403, detail=f"Model {model} not allowed for tenant")

# Check rate limit
if not check_rate_limit(x_tenant_id, config):
    raise HTTPException(status_code=429, detail="Rate limit exceeded")

# Check concurrent requests
if not check_concurrent(x_tenant_id, config):
    raise HTTPException(status_code=429, detail="Concurrent request limit exceeded")

# Check budget
if not check_budget(x_tenant_id, config):
    raise HTTPException(status_code=402, detail="Daily budget exceeded")

# Route to model endpoint
endpoint = MODEL_ENDPOINTS.get(model)
if not endpoint:
    raise HTTPException(status_code=404, detail=f"Model {model} not available")

# Track concurrent requests
concurrent_key = f"concurrent:{x_tenant_id}"
redis_client.incr(concurrent_key)

try:
    async with httpx.AsyncClient(timeout=120.0) as client:
        response = await client.post(
            f"{endpoint}/v1/chat/completions",
            json=body,
            headers={"Content-Type": "application/json"},
        )
        result = response.json()

        # Record usage
        usage = result.get("usage", {})
        record_usage(
            x_tenant_id, model,
            usage.get("prompt_tokens", 0),
            usage.get("completion_tokens", 0),
        )

        return result
finally:
    redis_client.decr(concurrent_key)

undefined

Rate Limiting with Envoy

使用Envoy进行速率限制

yaml

undefined

yaml

undefined

envoy-ratelimit.yaml

apiVersion: v1 kind: ConfigMap metadata: name: envoy-ratelimit-config namespace: llm-serving data: config.yaml: | domain: llm-gateway descriptors: # Per-tenant rate limits - key: tenant_id value: acme-corp rate_limit: unit: minute requests_per_unit: 300 - key: tenant_id value: startup-xyz rate_limit: unit: minute requests_per_unit: 60 - key: tenant_id value: internal-dev rate_limit: unit: minute requests_per_unit: 20

  # Global rate limit as safety net
  - key: global
    rate_limit:
      unit: second
      requests_per_unit: 100

undefined

apiVersion: v1 kind: ConfigMap metadata: name: envoy-ratelimit-config namespace: llm-serving data: config.yaml: | domain: llm-gateway descriptors: # Per-tenant rate limits - key: tenant_id value: acme-corp rate_limit: unit: minute requests_per_unit: 300 - key: tenant_id value: startup-xyz rate_limit: unit: minute requests_per_unit: 60 - key: tenant_id value: internal-dev rate_limit: unit: minute requests_per_unit: 20

  # Global rate limit as safety net
  - key: global
    rate_limit:
      unit: second
      requests_per_unit: 100

undefined

Billing Integration

账单集成

python

undefined

python

undefined

billing_export.py

"""Export tenant usage data for billing systems.""" import redis import json from datetime import datetime, timedelta from typing import Dict, List

redis_client = redis.Redis(host="redis", port=6379, decode_responses=True)

def generate_tenant_invoice(tenant_id: str, month: str) -> Dict: """Generate monthly invoice for a tenant.""" billing_key = f"billing:{tenant_id}:{month}" records = redis_client.lrange(billing_key, 0, -1)

usage_by_model = {}
total_cost = 0.0
total_requests = 0

for record_json in records:
    record = json.loads(record_json)
    model = record["model"]

    if model not in usage_by_model:
        usage_by_model[model] = {
            "requests": 0,
            "prompt_tokens": 0,
            "completion_tokens": 0,
            "cost_usd": 0.0,
        }

    usage_by_model[model]["requests"] += 1
    usage_by_model[model]["prompt_tokens"] += record["prompt_tokens"]
    usage_by_model[model]["completion_tokens"] += record["completion_tokens"]
    usage_by_model[model]["cost_usd"] += record["cost_usd"]

    total_cost += record["cost_usd"]
    total_requests += 1

return {
    "tenant_id": tenant_id,
    "billing_period": month,
    "generated_at": datetime.utcnow().isoformat(),
    "summary": {
        "total_requests": total_requests,
        "total_cost_usd": round(total_cost, 4),
    },
    "usage_by_model": usage_by_model,
}

def get_tenant_spend_today(tenant_id: str) -> float: """Get current day spend for budget alerts.""" key = f"spend:{tenant_id}:{datetime.utcnow().strftime('%Y-%m-%d')}" return float(redis_client.get(key) or 0)

undefined

"""Export tenant usage data for billing systems.""" import redis import json from datetime import datetime, timedelta from typing import Dict, List

redis_client = redis.Redis(host="redis", port=6379, decode_responses=True)

def generate_tenant_invoice(tenant_id: str, month: str) -> Dict: """Generate monthly invoice for a tenant.""" billing_key = f"billing:{tenant_id}:{month}" records = redis_client.lrange(billing_key, 0, -1)

usage_by_model = {}
total_cost = 0.0
total_requests = 0

for record_json in records:
    record = json.loads(record_json)
    model = record["model"]

    if model not in usage_by_model:
        usage_by_model[model] = {
            "requests": 0,
            "prompt_tokens": 0,
            "completion_tokens": 0,
            "cost_usd": 0.0,
        }

    usage_by_model[model]["requests"] += 1
    usage_by_model[model]["prompt_tokens"] += record["prompt_tokens"]
    usage_by_model[model]["completion_tokens"] += record["completion_tokens"]
    usage_by_model[model]["cost_usd"] += record["cost_usd"]

    total_cost += record["cost_usd"]
    total_requests += 1

return {
    "tenant_id": tenant_id,
    "billing_period": month,
    "generated_at": datetime.utcnow().isoformat(),
    "summary": {
        "total_requests": total_requests,
        "total_cost_usd": round(total_cost, 4),
    },
    "usage_by_model": usage_by_model,
}

def get_tenant_spend_today(tenant_id: str) -> float: """Get current day spend for budget alerts.""" key = f"spend:{tenant_id}:{datetime.utcnow().strftime('%Y-%m-%d')}" return float(redis_client.get(key) or 0)

undefined

Noisy-Neighbor Controls

噪声邻居控制

Per-tenant RPM/TPM limits
Concurrency caps and queue isolation
Fair scheduling with weighted priority classes
Backpressure and graceful degradation policies

yaml

undefined

租户级RPM/TPM限制
并发上限与队列隔离
基于加权优先级类别的公平调度
背压与优雅降级策略

yaml

undefined

priority-classes.yaml

apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tenant-enterprise value: 1000 globalDefault: false description: "Enterprise tenant workloads"

apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tenant-standard value: 500 globalDefault: false description: "Standard tenant workloads"

apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tenant-free value: 100 globalDefault: false description: "Free tier tenant workloads"

undefined

apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tenant-enterprise value: 1000 globalDefault: false description: "Enterprise tenant workloads"

apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tenant-standard value: 500 globalDefault: false description: "Standard tenant workloads"

apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tenant-free value: 100 globalDefault: false description: "Free tier tenant workloads"

undefined

Per-Tenant Monitoring

租户级监控

yaml

undefined

yaml

undefined

tenant-alerts.yaml

groups:

name: tenant-alerts rules:
- alert: TenantBudgetWarning expr: | llm_tenant_daily_spend_usd / llm_tenant_daily_budget_usd > 0.80 for: 5m labels: severity: warning annotations: summary: "Tenant {{ $labels.tenant }} at 80% of daily budget"
- alert: TenantRateLimitHitting expr: | rate(llm_rate_limit_rejections_total[5m]) > 1 for: 5m labels: severity: info annotations: summary: "Tenant {{ $labels.tenant }} hitting rate limits"
- alert: TenantErrorRateHigh expr: | rate(llm_tenant_errors_total[5m]) / rate(llm_tenant_requests_total[5m]) > 0.10 for: 5m labels: severity: warning annotations: summary: "Tenant {{ $labels.tenant }} error rate above 10%"

undefined

groups:

name: tenant-alerts rules:
- alert: TenantBudgetWarning expr: | llm_tenant_daily_spend_usd / llm_tenant_daily_budget_usd > 0.80 for: 5m labels: severity: warning annotations: summary: "Tenant {{ $labels.tenant }} at 80% of daily budget"
- alert: TenantRateLimitHitting expr: | rate(llm_rate_limit_rejections_total[5m]) > 1 for: 5m labels: severity: info annotations: summary: "Tenant {{ $labels.tenant }} hitting rate limits"
- alert: TenantErrorRateHigh expr: | rate(llm_tenant_errors_total[5m]) / rate(llm_tenant_requests_total[5m]) > 0.10 for: 5m labels: severity: warning annotations: summary: "Tenant {{ $labels.tenant }} error rate above 10%"

undefined

Security Baseline

安全基线

Encrypt data in transit and at rest.
Disallow cross-tenant cache leakage.
Restrict debug data access by role.
Audit all privileged administrative actions.

加密传输中与静态存储的数据。
禁止跨租户缓存泄漏。
按角色限制调试数据访问。
审计所有特权管理操作。

Operational Runbook

运维手册

Onboard tenant with policy template.
Issue virtual key and quota profile.
Validate observability and billing tags.
Run tenant-specific load/safety tests.
Enable production traffic with canary limits.

使用策略模板接入租户。
发放虚拟密钥和配额配置文件。
验证可观测性和账单标签。
运行租户专属的负载/安全测试。
启用金丝雀限制的生产流量。

Troubleshooting

故障排查

Symptom	Check	Fix
Tenant getting 429 errors	Rate limit counters in Redis	Increase RPM/TPM limits or upgrade tier
One tenant slowing others	Concurrent request counts per tenant	Reduce concurrency cap for offending tenant
Billing data missing	Redis billing keys and export job logs	Check billing export CronJob and Redis connectivity
Tenant cannot access model	Tenant config in ConfigMap	Add model to `models_allowed` list
Cross-tenant data leakage	Cache key prefixes and namespace isolation	Ensure cache keys include tenant_id prefix
Budget alerts not firing	Prometheus scrape targets and alert rules	Verify metric export and Alertmanager config

症状	检查项	修复方案
租户收到429错误	Redis中的速率限制计数器	提高RPM/TPM限制或升级租户层级
单个租户拖慢其他租户	各租户的并发请求数	降低违规租户的并发上限
账单数据缺失	Redis账单密钥和导出作业日志	检查账单导出CronJob和Redis连接性
租户无法访问模型	ConfigMap中的租户配置	将模型添加到 `models_allowed` 列表
跨租户数据泄漏	缓存键前缀和命名空间隔离	确保缓存键包含tenant_id前缀
预算警报未触发	Prometheus采集目标和警报规则	验证指标导出和Alertmanager配置

multi-tenant-llm-hosting

Original

Translation

Multi-Tenant LLM Hosting

多租户LLM托管

When to Use This Skill

何时使用该技能

Prerequisites

前置条件

Isolation Model

隔离模型

vLLM Multi-Model Serving

vLLM多模型服务

vllm-deployment.yaml - Multi-model serving with vLLM

vllm-deployment.yaml - Multi-model serving with vLLM

Per-Tenant Quota Configuration

租户级配额配置

tenant-quotas-configmap.yaml

tenant-quotas-configmap.yaml

Namespace Isolation for High-Risk Tenants

高风险租户的命名空间隔离

tenant-namespace.yaml

tenant-namespace.yaml

apiVersion: v1 kind: Namespace metadata: name: tenant-acme-corp labels: tenant: acme-corp isolation: strict

apiVersion: v1 kind: Namespace metadata: name: tenant-acme-corp labels: tenant: acme-corp isolation: strict

Request Routing and Rate Limiting

请求路由与速率限制

gateway_router.py

gateway_router.py

Load tenant config

Load tenant config

Rate Limiting with Envoy

使用Envoy进行速率限制

envoy-ratelimit.yaml

envoy-ratelimit.yaml

Billing Integration

账单集成

billing_export.py

billing_export.py

Noisy-Neighbor Controls

噪声邻居控制

priority-classes.yaml

priority-classes.yaml

apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tenant-enterprise value: 1000 globalDefault: false description: "Enterprise tenant workloads"

apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tenant-standard value: 500 globalDefault: false description: "Standard tenant workloads"

apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tenant-enterprise value: 1000 globalDefault: false description: "Enterprise tenant workloads"

apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: tenant-standard value: 500 globalDefault: false description: "Standard tenant workloads"

Per-Tenant Monitoring

租户级监控

tenant-alerts.yaml

tenant-alerts.yaml

Security Baseline

安全基线

Operational Runbook

运维手册

Troubleshooting

故障排查

Related Skills

相关技能