gke-tpu-metrics-monitoring

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

GKE TPU Metrics Monitoring Guide

GKE TPU指标监控指南

This skill enables the agent to monitor GKE TPU workloads, nodes, and node pools using GKE system metrics. It helps diagnose if workload interruptions or performance issues are caused by underlying infrastructure.

本技能支持Agent使用GKE系统指标监控GKE TPU工作负载、节点及节点池，帮助诊断工作负载中断或性能问题是否由底层基础设施导致。

Step 0: Mandatory Context

步骤0：必备上下文

Independently gather required context (such as cluster details or node pool names) using available GKE and Cloud tools, or use the provided

{variable}

placeholders:

```
{project_id}
```
: The GCP Project ID.
```
{cluster_name}
```
: The GKE Cluster Name.
```
{location}
```
: The GKE Cluster Location (region or zone).
```
{node_name}
```
: (Optional) The name of the specific GKE node.
```
{node_pool_name}
```
: (Optional) The name of the GKE node pool.

使用可用的GKE及云工具自行收集所需上下文（如集群详情或节点池名称），或使用提供的

{variable}

占位符：

```
{project_id}
```
：GCP项目ID。
```
{cluster_name}
```
：GKE集群名称。
```
{location}
```
：GKE集群位置（区域或可用区）。
```
{node_name}
```
：（可选）特定GKE节点的名称。
```
{node_pool_name}
```
：（可选）GKE节点池的名称。

Diagnostic Steps

诊断步骤

Step 1: Verify TPU Runtime Metrics Configuration [Low Risk] [Auto]

步骤1：验证TPU运行时指标配置 [低风险] [自动]

Before analyzing runtime metrics, verify that the workload is configured to export them.

Action: Verify that the Pod specification for the TPU workload includes:
- ```
containerPort: 8431
```
- JAX version
```
0.4.14
```
  or later (if using JAX).
- GKE version is
```
1.27.4-gke.900
```
  or later.
- GKE System Metrics are enabled on the cluster.

分析运行时指标前，需验证工作负载已配置为导出指标。

操作：验证TPU工作负载的Pod规范包含以下内容：
- ```
containerPort: 8431
```
- JAX版本为
```
0.4.14
```
  或更高版本（若使用JAX）。
- GKE版本为
```
1.27.4-gke.900
```
  或更高版本。
- 集群已启用GKE系统指标。

Step 2: Monitor TPU Runtime Metrics [Low Risk] [Auto]

步骤2：监控TPU运行时指标 [低风险] [自动]

If configured correctly, the following metrics are available in Cloud Monitoring (monitored resources

k8s_node

and

k8s_container

Container Metrics:
- ```
kubernetes.io/container/accelerator/duty_cycle
```
  : Percentage of time over the past sampling period (60 seconds) during which the TensorCores were actively processing on a TPU chip.
- ```
kubernetes.io/container/accelerator/memory_used
```
  : Amount of accelerator memory allocated in bytes.
- ```
kubernetes.io/container/accelerator/memory_total
```
  : Total accelerator memory in bytes.

Node Metrics:

kubernetes.io/node/accelerator/duty_cycle

kubernetes.io/node/accelerator/memory_used

kubernetes.io/node/accelerator/memory_total

若配置正确，以下指标可在Cloud Monitoring中查看（监控资源为

k8s_node

和

k8s_container

）：

容器指标：
- ```
kubernetes.io/container/accelerator/duty_cycle
```
  ：过去采样周期（60秒）内，TPU芯片上TensorCore处于活跃处理状态的时间占比。
- ```
kubernetes.io/container/accelerator/memory_used
```
  ：已分配的加速器内存大小（字节）。
- ```
kubernetes.io/container/accelerator/memory_total
```
  ：加速器总内存大小（字节）。

节点指标：

kubernetes.io/node/accelerator/duty_cycle

kubernetes.io/node/accelerator/memory_used

kubernetes.io/node/accelerator/memory_total

Step 3: Check Node Status Condition [Low Risk] [Auto]

步骤3：检查节点状态条件 [低风险] [自动]

Query the status condition of GKE nodes (GKE version

1.32.1-gke.1357001

or later).

PromQL Query (Check if a specific node is Ready):

promql

kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", node_name="{node_name}", condition="Ready", status="True"}

PromQL Query (List nodes with non-Ready conditions that are True):

promql

kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition!="Ready", status="True"}

PromQL Query (List nodes that are NOT Ready):

promql

kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition="Ready", status="False"}

PromQL Query (Fleet-wide node status):

promql

avg by (condition,status)(avg_over_time(kubernetes_io:node_status_condition{monitored_resource="k8s_node"}[5m]))

查询GKE节点的状态条件（GKE版本

1.32.1-gke.1357001

或更高版本）。

PromQL查询（检查特定节点是否就绪）：

promql

kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", node_name="{node_name}", condition="Ready", status="True"}

PromQL查询（列出存在非就绪状态条件的节点）：

promql

kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition!="Ready", status="True"}

PromQL查询（列出未就绪的节点）：

promql

kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition="Ready", status="False"}

PromQL查询（集群范围内节点状态）：

promql

avg by (condition,status)(avg_over_time(kubernetes_io:node_status_condition{monitored_resource="k8s_node"}[5m]))

Step 4: Check Node Pool Status [Low Risk] [Auto]

步骤4：检查节点池状态 [低风险] [自动]

Query the status of multi-host TPU node pools.

PromQL Query (Verify if a specific node pool is Running):

promql

kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}", node_pool_name="{node_pool_name}", status="Running"}

PromQL Query (Monitor node pools grouped by status):

promql

count by (status)(count_over_time(kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool"}[5m]))

Possible statuses:

Provisioning

Running

Error

Reconciling

Stopping

查询多主机TPU节点池的状态。

PromQL查询（验证特定节点池是否处于运行状态）：

promql

kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}", node_pool_name="{node_pool_name}", status="Running"}

PromQL查询（按状态分组监控节点池）：

promql

count by (status)(count_over_time(kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool"}[5m]))

可能的状态:

Provisioning

Running

Error

Reconciling

Stopping

Step 5: Check Node Pool Availability [Low Risk] [Auto]

步骤5：检查节点池可用性 [低风险] [自动]

Query if all nodes in a multi-host TPU node pool are available.

PromQL Query (Check availability over time):

promql

avg by (node_pool_name)(avg_over_time(kubernetes_io:node_pool_multi_host_available{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[5m]))

Value:

(True, all nodes available) or

(False, some nodes unavailable).

查询多主机TPU节点池中的所有节点是否可用。

PromQL查询（检查一段时间内的可用性）：

promql

avg by (node_pool_name)(avg_over_time(kubernetes_io:node_pool_multi_host_available{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[5m]))

取值:

（是，所有节点可用）或

（否，部分节点不可用）。

Step 6: Analyze Node Interruptions [Low Risk] [Auto]

步骤6：分析节点中断情况 [低风险] [自动]

Query the count of interruptions for GKE nodes.

PromQL Query (Breakdown of interruptions and causes):

promql

sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node"}[5m]))

Interruption Types:

TerminationEvent

MaintenanceEvent

PreemptionEvent

. Interruption Reasons:

HostError

Eviction

AutoRepair

PromQL Query (Filter for Host Maintenance events):

promql

sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", interruption_reason="HW/SW Maintenance"}[5m]))

PromQL Query (Interruption count aggregated by node pool):

promql

sum by (node_pool_name,interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_pool_interruption_count{monitored_resource="k8s_node_pool", interruption_reason="HW/SW Maintenance", node_pool_name="{node_pool_name}"}[5m]))

查询GKE节点的中断次数。

PromQL查询（中断类型及原因细分）：

promql

sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node"}[5m]))

中断类型:

TerminationEvent

MaintenanceEvent

PreemptionEvent

. 中断原因:

HostError

Eviction

AutoRepair

PromQL查询（筛选主机维护事件）：

promql

sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", interruption_reason="HW/SW Maintenance"}[5m]))

PromQL查询（按节点池聚合中断次数）：

promql

sum by (node_pool_name,interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_pool_interruption_count{monitored_resource="k8s_node_pool", interruption_reason="HW/SW Maintenance", node_pool_name="{node_pool_name}"}[5m]))

Step 7: Calculate Recovery and Interruption Metrics [Low Risk] [Auto]

步骤7：计算恢复与中断指标 [低风险] [自动]

Calculate Mean Time to Recovery (MTTR) and Mean Time Between Interruptions (MTBI) over the last 7 days.

PromQL Query (MTTR - Mean Time to Recovery):

promql

sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_sum{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_count{monitored_resource="k8s_node_pool",cluster_name="{cluster_name}"}[7d]))

PromQL Query (MTBI - Mean Time Between Interruptions):

promql

sum(count_over_time(kubernetes_io:node_memory_total_bytes{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d]))

计算过去7天的平均恢复时间（MTTR）和平均中断间隔时间（MTBI）。

PromQL查询（MTTR - 平均恢复时间）：

promql

sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_sum{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_count{monitored_resource="k8s_node_pool",cluster_name="{cluster_name}"}[7d]))

PromQL查询（MTBI - 平均中断间隔时间）：

promql

sum(count_over_time(kubernetes_io:node_memory_total_bytes{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d]))

Step 8: Monitor TPU Host Metrics [Low Risk] [Auto]

步骤8：监控TPU主机指标 [低风险] [自动]

For GKE version

1.28.1-gke.1066000

or later, monitor TPU host performance.

Container Metrics:
- ```
kubernetes.io/container/accelerator/tensorcore_utilization
```
  : Current percentage of the TensorCore that is utilized.
- ```
kubernetes.io/container/accelerator/memory_bandwidth_utilization
```
  : Current percentage of the accelerator memory bandwidth that is being used.

Node Metrics:

kubernetes.io/node/accelerator/tensorcore_utilization

kubernetes.io/node/accelerator/memory_bandwidth_utilization

对于GKE版本

1.28.1-gke.1066000

或更高版本，监控TPU主机性能。

容器指标：

kubernetes.io/container/accelerator/tensorcore_utilization

：当前TensorCore的利用率百分比。

kubernetes.io/container/accelerator/memory_bandwidth_utilization

：当前加速器内存带宽的使用率百分比。

节点指标：

kubernetes.io/node/accelerator/tensorcore_utilization

kubernetes.io/node/accelerator/memory_bandwidth_utilization