Back to Details

gke-ai-troubleshooting-tpu-metrics-monitoring

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

GKE TPU Metrics Monitoring Guide

GKE TPU指标监控指南

This skill enables the agent to monitor GKE TPU workloads, nodes, and node pools using GKE system metrics. It helps diagnose if workload interruptions or performance issues are caused by underlying infrastructure.
本技能支持Agent使用GKE系统指标监控GKE TPU工作负载、节点及节点池，有助于诊断工作负载中断或性能问题是否由底层基础设施导致。

Step 0: Mandatory Context

步骤0：必备上下文

Independently gather required context (such as cluster details or node pool names) using available GKE and Cloud tools, or use the provided 
{variable}
placeholders:
  • {project_id}
    : The GCP Project ID.
  • {cluster_name}
    : The GKE Cluster Name.
  • {location}
    : The GKE Cluster Location (region or zone).
  • {node_name}
    : (Optional) The name of the specific GKE node.
  • {node_pool_name}
    : (Optional) The name of the GKE node pool.
使用可用的GKE和云工具自行收集所需上下文（如集群详情或节点池名称），或使用提供的
{variable}
占位符：
  • {project_id}
    ：GCP项目ID。
  • {cluster_name}
    ：GKE集群名称。
  • {location}
    ：GKE集群位置（区域或可用区）。
  • {node_name}
    ：（可选）特定GKE节点的名称。
  • {node_pool_name}
    ：（可选）GKE节点池的名称。

Diagnostic Steps

诊断步骤

Step 1: Verify TPU Runtime Metrics Configuration [Low Risk] [Auto]

步骤1：验证TPU运行时指标配置 [低风险] [自动]

Before analyzing runtime metrics, verify that the workload is configured to export them. This ensures the cluster and container environment are set up for automated metric scraping and visibility into accelerator health.
  • Action: Verify that the Pod specification and cluster meet the following prerequisites: 
    • containerPort: 8431
      exposed on the TPU container (required for Prometheus metric scraping).
    • JAX version 
      0.4.14
      or later if using JAX (earlier versions do not export runtime metrics).
    • GKE version is 
      1.27.4-gke.900
      or later (required for TPU runtime metric support).
    • GKE System Metrics are enabled on the cluster (required for Cloud Monitoring ingestion).
分析运行时指标前，需验证工作负载已配置为导出指标。这确保集群和容器环境已设置好自动指标采集，可查看加速器健康状态。
  • 操作：验证Pod规格和集群满足以下前提条件：
    • TPU容器暴露
      containerPort: 8431
      （Prometheus指标采集所需）。
    • 若使用JAX，JAX版本为
      0.4.14
      或更高（早期版本不导出运行时指标）。
    • GKE版本为
      1.27.4-gke.900
      或更高（支持TPU运行时指标所需）。
    • 集群已启用GKE系统指标（Cloud Monitoring数据导入所需）。

Step 2: Monitor TPU Runtime Metrics [Low Risk] [Auto]

步骤2：监控TPU运行时指标 [低风险] [自动]

If configured correctly, the following metrics are available in Cloud Monitoring (monitored resources 
k8s_node
and 
k8s_container
):
  • Container Metrics: 
    • kubernetes.io/container/accelerator/duty_cycle
      : Percentage of time over the past sampling period (60 seconds) during which the TensorCores were actively processing on a TPU chip.
    • kubernetes.io/container/accelerator/memory_used
      : Amount of accelerator memory allocated in bytes.
    • kubernetes.io/container/accelerator/memory_total
      : Total accelerator memory in bytes.
  • Node Metrics: 
    • kubernetes.io/node/accelerator/duty_cycle
    • kubernetes.io/node/accelerator/memory_used
    • kubernetes.io/node/accelerator/memory_total
配置正确的情况下，以下指标可在Cloud Monitoring中查看（监控资源为
k8s_node
和
k8s_container
）：
  • 容器指标： 
    • kubernetes.io/container/accelerator/duty_cycle
      ：过去采样周期（60秒）内，TPU芯片上TensorCore处于活跃处理状态的时间占比。
    • kubernetes.io/container/accelerator/memory_used
      ：已分配的加速器内存字节数。
    • kubernetes.io/container/accelerator/memory_total
      ：加速器总内存字节数。
  • 节点指标： 
    • kubernetes.io/node/accelerator/duty_cycle
    • kubernetes.io/node/accelerator/memory_used
    • kubernetes.io/node/accelerator/memory_total

Step 3: Check Node Status Condition [Low Risk] [Auto]

步骤3：检查节点状态条件 [低风险] [自动]

Query the status condition of GKE nodes (GKE version 
1.32.1-gke.1357001
or later).
  • PromQL Query (Check if a specific node is Ready):
    promql
    kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", node_name="{node_name}", condition="Ready", status="True"}
  • PromQL Query (List nodes with non-Ready conditions that are True):
    promql
    kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition!="Ready", status="True"}
  • PromQL Query (List nodes that are NOT Ready):
    promql
    kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition="Ready", status="False"}
  • PromQL Query (Fleet-wide node status):
    promql
    avg by (condition,status)(avg_over_time(kubernetes_io:node_status_condition{monitored_resource="k8s_node"}[5m]))
查询GKE节点的状态条件（GKE版本
1.32.1-gke.1357001
或更高）。
  • PromQL查询（检查特定节点是否就绪）：
    promql
    kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", node_name="{node_name}", condition="Ready", status="True"}
  • PromQL查询（列出存在非就绪且状态为True的节点）：
    promql
    kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition!="Ready", status="True"}
  • PromQL查询（列出未就绪的节点）：
    promql
    kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition="Ready", status="False"}
  • PromQL查询（集群范围内节点状态）：
    promql
    avg by (condition,status)(avg_over_time(kubernetes_io:node_status_condition{monitored_resource="k8s_node"}[5m]))

Step 4: Check Node Pool Status [Low Risk] [Auto]

步骤4：检查节点池状态 [低风险] [自动]

Query the status of multi-host TPU node pools.
  • PromQL Query (Verify if a specific node pool is Running):
    promql
    kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}", node_pool_name="{node_pool_name}", status="Running"}
  • PromQL Query (Monitor node pools grouped by status):
    promql
    count by (status)(count_over_time(kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool"}[5m]))
    Possible statuses: 
    Provisioning
    , 
    Running
    , 
    Error
    , 
    Reconciling
    , 
    Stopping
    .
查询多主机TPU节点池的状态。
  • PromQL查询（验证特定节点池是否处于运行状态）：
    promql
    kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}", node_pool_name="{node_pool_name}", status="Running"}
  • PromQL查询（按状态分组监控节点池）：
    promql
    count by (status)(count_over_time(kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool"}[5m]))
    可能的状态: 
    Provisioning
    , 
    Running
    , 
    Error
    , 
    Reconciling
    , 
    Stopping
    .

Step 5: Check Node Pool Availability [Low Risk] [Auto]

步骤5：检查节点池可用性 [低风险] [自动]

Query if all nodes in a multi-host TPU node pool are available.
  • PromQL Query (Check availability over time):
    promql
    avg by (node_pool_name)(avg_over_time(kubernetes_io:node_pool_multi_host_available{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[5m]))
    Value: 
    1
    (True, all nodes available) or 
    0
    (False, some nodes unavailable).
查询多主机TPU节点池中的所有节点是否可用。
  • PromQL查询（检查一段时间内的可用性）：
    promql
    avg by (node_pool_name)(avg_over_time(kubernetes_io:node_pool_multi_host_available{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[5m]))
    数值: 
    1
    （是，所有节点可用）或
    0
    （否，部分节点不可用）。

Step 6: Analyze Node Interruptions [Low Risk] [Auto]

步骤6：分析节点中断 [低风险] [自动]

Query the count of interruptions for GKE nodes.
  • PromQL Query (Breakdown of interruptions and causes):
    promql
    sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node"}[5m]))
    Interruption Types: 
    TerminationEvent
    , 
    MaintenanceEvent
    , 
    PreemptionEvent
    . Interruption Reasons: 
    HostError
    , 
    Eviction
    , 
    AutoRepair
    .
  • PromQL Query (Filter for Host Maintenance events):
    promql
    sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", interruption_reason="HW/SW Maintenance"}[5m]))
  • PromQL Query (Interruption count aggregated by node pool):
    promql
    sum by (node_pool_name,interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_pool_interruption_count{monitored_resource="k8s_node_pool", interruption_reason="HW/SW Maintenance", node_pool_name="{node_pool_name}"}[5m]))
查询GKE节点的中断次数。
  • PromQL查询（中断类型及原因细分）：
    promql
    sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node"}[5m]))
    中断类型: 
    TerminationEvent
    , 
    MaintenanceEvent
    , 
    PreemptionEvent
    . 中断原因: 
    HostError
    , 
    Eviction
    , 
    AutoRepair
    .
  • PromQL查询（筛选主机维护事件）：
    promql
    sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", interruption_reason="HW/SW Maintenance"}[5m]))
  • PromQL查询（按节点池聚合中断次数）：
    promql
    sum by (node_pool_name,interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_pool_interruption_count{monitored_resource="k8s_node_pool", interruption_reason="HW/SW Maintenance", node_pool_name="{node_pool_name}"}[5m]))

Step 7: Calculate Recovery and Interruption Metrics [Low Risk] [Auto]

步骤7：计算恢复与中断指标 [低风险] [自动]

Calculate Mean Time to Recovery (MTTR) and Mean Time Between Interruptions (MTBI) over the last 7 days.
  • PromQL Query (MTTR - Mean Time to Recovery):
    promql
    sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_sum{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_count{monitored_resource="k8s_node_pool",cluster_name="{cluster_name}"}[7d]))
  • PromQL Query (MTBI - Mean Time Between Interruptions):
    promql
    sum(count_over_time(kubernetes_io:node_memory_total_bytes{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d]))
计算过去7天的平均恢复时间（MTTR）和平均中断间隔时间（MTBI）。
  • PromQL查询（MTTR - 平均恢复时间）：
    promql
    sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_sum{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_count{monitored_resource="k8s_node_pool",cluster_name="{cluster_name}"}[7d]))
  • PromQL查询（MTBI - 平均中断间隔时间）：
    promql
    sum(count_over_time(kubernetes_io:node_memory_total_bytes{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d]))

Step 8: Monitor TPU Host Metrics [Low Risk] [Auto]

步骤8：监控TPU主机指标 [低风险] [自动]

For GKE version 
1.28.1-gke.1066000
or later, monitor TPU host performance.
  • Container Metrics: 
    • kubernetes.io/container/accelerator/tensorcore_utilization
      : Current percentage of the TensorCore that is utilized.
    • kubernetes.io/container/accelerator/memory_bandwidth_utilization
      : Current percentage of the accelerator memory bandwidth that is being used.
  • Node Metrics: 
    • kubernetes.io/node/accelerator/tensorcore_utilization
    • kubernetes.io/node/accelerator/memory_bandwidth_utilization
对于GKE版本
1.28.1-gke.1066000
或更高版本，监控TPU主机性能。
  • 容器指标： 
    • kubernetes.io/container/accelerator/tensorcore_utilization
      ：当前TensorCore的使用率百分比。
    • kubernetes.io/container/accelerator/memory_bandwidth_utilization
      ：当前加速器内存带宽的使用率百分比。
  • 节点指标： 
    • kubernetes.io/node/accelerator/tensorcore_utilization
    • kubernetes.io/node/accelerator/memory_bandwidth_utilization