LLM Inference Scaling

LLM推理扩缩

Scale LLM inference horizontally on Kubernetes with GPU-aware autoscaling, request queuing, and cost-efficient spot instance strategies.

在Kubernetes上通过GPU感知的自动扩缩、请求排队和高性价比的抢占式实例策略，实现LLM推理的水平扩缩。

When to Use This Skill

何时使用该技能

Use this skill when:

LLM API traffic is unpredictable and you need to scale up/down automatically
Managing a fleet of vLLM or TGI inference pods on Kubernetes
Reducing inference costs with spot/preemptible GPU instances
Implementing queue-based autoscaling for batch inference jobs
Building a multi-model serving platform that shares GPU resources

在以下场景中使用该技能：

LLM API流量不可预测，需要自动进行扩缩容
在Kubernetes上管理vLLM或TGI推理Pod集群
通过抢占式GPU实例降低推理成本
为批量推理任务实现基于队列的自动扩缩
构建共享GPU资源的多模型服务平台

Prerequisites

前提条件

Kubernetes cluster with GPU nodes (NVIDIA operator installed)
KEDA (Kubernetes Event-Driven Autoscaler) installed
Prometheus with GPU metrics (
```
dcgm-exporter
```
or
```
gpu-operator
```
)
Helm 3+ for chart deployments

配备GPU节点的Kubernetes集群（已安装NVIDIA operator）
已安装KEDA（Kubernetes事件驱动自动扩缩器）
带有GPU指标的Prometheus（
```
dcgm-exporter
```
或
```
gpu-operator
```
）
用于Chart部署的Helm 3及以上版本

GPU Node Setup

GPU节点配置

bash

undefined

bash

undefined

Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM)

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm repo update

helm install gpu-operator nvidia/gpu-operator
--namespace gpu-operator
--create-namespace
--set driver.enabled=true
--set dcgm.enabled=true
--set devicePlugin.enabled=true

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm repo update

helm install gpu-operator nvidia/gpu-operator
--namespace gpu-operator
--create-namespace
--set driver.enabled=true
--set dcgm.enabled=true
--set devicePlugin.enabled=true

Verify GPU nodes are recognized

kubectl get nodes -l nvidia.com/gpu.present=true kubectl describe node <gpu-node> | grep nvidia

undefined

kubectl get nodes -l nvidia.com/gpu.present=true kubectl describe node <gpu-node> | grep nvidia

undefined

vLLM Deployment with GPU Resources

带有GPU资源的vLLM部署

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama-8b
  labels:
    app: vllm
    model: llama-3.1-8b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
      model: llama-3.1-8b
  template:
    metadata:
      labels:
        app: vllm
        model: llama-3.1-8b
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - "--model"
        - "meta-llama/Llama-3.1-8B-Instruct"
        - "--tensor-parallel-size"
        - "1"
        - "--gpu-memory-utilization"
        - "0.90"
        - "--max-num-seqs"
        - "128"
        resources:
          requests:
            nvidia.com/gpu: "1"
            memory: "20Gi"
            cpu: "4"
          limits:
            nvidia.com/gpu: "1"
            memory: "24Gi"
            cpu: "8"
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama-8b
  labels:
    app: vllm
    model: llama-3.1-8b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
      model: llama-3.1-8b
  template:
    metadata:
      labels:
        app: vllm
        model: llama-3.1-8b
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - "--model"
        - "meta-llama/Llama-3.1-8B-Instruct"
        - "--tensor-parallel-size"
        - "1"
        - "--gpu-memory-utilization"
        - "0.90"
        - "--max-num-seqs"
        - "128"
        resources:
          requests:
            nvidia.com/gpu: "1"
            memory: "20Gi"
            cpu: "4"
          limits:
            nvidia.com/gpu: "1"
            memory: "24Gi"
            cpu: "8"
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token

KEDA Autoscaling on Prometheus Metrics

基于Prometheus指标的KEDA自动扩缩

yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaledobject
spec:
  scaleTargetRef:
    name: vllm-llama-8b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300          # 5 min before scale-down
  pollingInterval: 15
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring:9090
      metricName: vllm_num_requests_waiting
      threshold: "10"           # scale up if >10 requests waiting
      query: |
        sum(vllm:num_requests_waiting{deployment="vllm-llama-8b"})
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring:9090
      metricName: vllm_gpu_cache_usage
      threshold: "0.8"          # scale up if KV cache >80% full
      query: |
        avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama-8b"})

yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaledobject
spec:
  scaleTargetRef:
    name: vllm-llama-8b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300          # 5 min before scale-down
  pollingInterval: 15
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring:9090
      metricName: vllm_num_requests_waiting
      threshold: "10"           # scale up if >10 requests waiting
      query: |
        sum(vllm:num_requests_waiting{deployment="vllm-llama-8b"})
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring:9090
      metricName: vllm_gpu_cache_usage
      threshold: "0.8"          # scale up if KV cache >80% full
      query: |
        avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama-8b"})

Queue-Based Scaling (Redis + KEDA)

基于队列的扩缩（Redis + KEDA）

yaml

undefined

yaml

undefined

ScaledJob for async batch inference

apiVersion: keda.sh/v1alpha1 kind: ScaledJob metadata: name: llm-batch-inference spec: jobTargetRef: template: spec: containers: - name: inference-worker image: myapp/inference-worker:latest env: - name: REDIS_URL value: redis://redis:6379 - name: QUEUE_NAME value: inference-jobs restartPolicy: OnFailure minReplicaCount: 0 maxReplicaCount: 20 pollingInterval: 5 successfulJobsHistoryLimit: 3 triggers:

type: redis metadata: address: redis:6379 listName: inference-jobs listLength: "5" # 1 worker per 5 queued jobs

undefined

apiVersion: keda.sh/v1alpha1 kind: ScaledJob metadata: name: llm-batch-inference spec: jobTargetRef: template: spec: containers: - name: inference-worker image: myapp/inference-worker:latest env: - name: REDIS_URL value: redis://redis:6379 - name: QUEUE_NAME value: inference-jobs restartPolicy: OnFailure minReplicaCount: 0 maxReplicaCount: 20 pollingInterval: 5 successfulJobsHistoryLimit: 3 triggers:

type: redis metadata: address: redis:6379 listName: inference-jobs listLength: "5" # 1 worker per 5 queued jobs

undefined

Spot Instance Strategy

抢占式实例策略

yaml

undefined

yaml

undefined

Mixed node pool: on-demand + spot GPUs

apiVersion: v1 kind: ConfigMap metadata: name: cluster-autoscaler-priority-config data: priorities: | 10: # low priority = prefer - .spot. 50: - .on-demand.

Node affinity for spot with on-demand fallback

spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 80 preference: matchExpressions: - key: node.kubernetes.io/lifecycle operator: In values: [spot] - weight: 20 preference: matchExpressions: - key: node.kubernetes.io/lifecycle operator: In values: [on-demand]

undefined

spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 80 preference: matchExpressions: - key: node.kubernetes.io/lifecycle operator: In values: [spot] - weight: 20 preference: matchExpressions: - key: node.kubernetes.io/lifecycle operator: In values: [on-demand]

undefined

Cluster Autoscaler for GPU Nodes

GPU节点的集群自动扩缩器

bash

undefined

bash

undefined

AWS EKS — enable cluster autoscaler for GPU node group

helm install cluster-autoscaler autoscaler/cluster-autoscaler
--namespace kube-system
--set autoDiscovery.clusterName=my-cluster
--set awsRegion=us-east-1
--set rbac.serviceAccount.annotations."eks.amazonaws.com/role-arn"=arn:aws:iam::ACCOUNT:role/ClusterAutoscalerRole
--set extraArgs.skip-nodes-with-local-storage=false
--set extraArgs.expander=least-waste

Annotate GPU node group for autoscaler

kubectl annotate node <node>
cluster-autoscaler.kubernetes.io/safe-to-evict="false"

undefined

kubectl annotate node <node>
cluster-autoscaler.kubernetes.io/safe-to-evict="false"

undefined

Scaling Metrics to Monitor

需监控的扩缩指标

bash

undefined

bash

undefined

Prometheus queries for scaling decisions

Requests waiting in vLLM queue

sum(vllm:num_requests_waiting) by (model)

GPU KV cache utilization (>80% = bottleneck)

avg(vllm:gpu_cache_usage_perc) by (pod)

Tokens per second throughput

sum(rate(vllm:generation_tokens_total[5m])) by (model)

P99 time-to-first-token

histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))

undefined

histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))

undefined

Common Issues

常见问题

Issue	Cause	Fix
Pods stuck in `Pending`	No GPU nodes available	Check cluster autoscaler logs; verify node group limits
Scale-up too slow	Cluster autoscaler delay + model load time	Pre-warm replicas; increase `minReplicaCount`
GPU fragmentation	Multiple small models on large GPUs	Use MIG partitioning or consolidate model sizes
Spot eviction causes errors	Spot instance reclamation	Add `PodDisruptionBudget` ; use graceful shutdown
KEDA not scaling	Prometheus query returns no data	Test query in Prometheus UI first

问题	原因	解决方法
Pod 卡在 `Pending` 状态	无可用GPU节点	检查集群自动扩缩器日志；验证节点组限制
扩容速度过慢	集群自动扩缩器延迟 + 模型加载时间	预预热副本；提高 `minReplicaCount`
GPU碎片化	大型GPU上运行多个小型模型	使用MIG分区或整合模型规格
抢占式实例回收导致错误	抢占式实例被回收	添加 `PodDisruptionBudget` ；使用优雅关闭
KEDA未执行扩缩	Prometheus查询无返回数据	先在Prometheus UI中测试查询

Best Practices

最佳实践

Set
```
minReplicaCount: 1
```
to avoid cold starts; scale to 0 only for batch jobs.
Use
```
PodDisruptionBudget
```
with
```
minAvailable: 1
```
to survive spot evictions.
Pre-pull model weights into a shared PVC to speed up pod startup by 5–10×.
Separate model families across node pools (A10G for 7B, A100 for 70B).
Use Kubernetes VPA for CPU/memory right-sizing alongside KEDA for replica count.

设置
```
minReplicaCount: 1
```
避免冷启动；仅针对批量任务缩容至0。
使用
```
PodDisruptionBudget
```
并设置
```
minAvailable: 1
```
，以应对抢占式实例回收。
将模型权重预拉取到共享PVC中，可将Pod启动速度提升5-10倍。
按模型系列划分节点池（A10G用于7B模型，A100用于70B模型）。
结合KEDA进行副本数管理，同时使用Kubernetes VPA优化CPU/内存配置。

llm-inference-scaling

Original

Translation

LLM Inference Scaling

LLM推理扩缩

When to Use This Skill

何时使用该技能

Prerequisites

前提条件

GPU Node Setup

GPU节点配置

Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM)

Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM)

Verify GPU nodes are recognized

Verify GPU nodes are recognized

vLLM Deployment with GPU Resources

带有GPU资源的vLLM部署

KEDA Autoscaling on Prometheus Metrics

基于Prometheus指标的KEDA自动扩缩

Queue-Based Scaling (Redis + KEDA)

基于队列的扩缩（Redis + KEDA）

ScaledJob for async batch inference

ScaledJob for async batch inference

Spot Instance Strategy

抢占式实例策略

Mixed node pool: on-demand + spot GPUs

Mixed node pool: on-demand + spot GPUs

apiVersion: v1 kind: ConfigMap metadata: name: cluster-autoscaler-priority-config data: priorities: | 10: # low priority = prefer - .spot. 50: - .on-demand.

apiVersion: v1 kind: ConfigMap metadata: name: cluster-autoscaler-priority-config data: priorities: | 10: # low priority = prefer - .spot. 50: - .on-demand.

Node affinity for spot with on-demand fallback

Node affinity for spot with on-demand fallback

Cluster Autoscaler for GPU Nodes

GPU节点的集群自动扩缩器

AWS EKS — enable cluster autoscaler for GPU node group

AWS EKS — enable cluster autoscaler for GPU node group

Annotate GPU node group for autoscaler

Annotate GPU node group for autoscaler

Scaling Metrics to Monitor

需监控的扩缩指标

Prometheus queries for scaling decisions

Prometheus queries for scaling decisions

Requests waiting in vLLM queue

Requests waiting in vLLM queue

GPU KV cache utilization (>80% = bottleneck)

GPU KV cache utilization (>80% = bottleneck)

Tokens per second throughput

Tokens per second throughput

P99 time-to-first-token

P99 time-to-first-token

Common Issues

常见问题

Best Practices

最佳实践

Related Skills

相关技能