gke-ai-troubleshooting-tpu-vbar-oom

Original：🇺🇸 English

Translated

1 scripts

Diagnoses and prevents vbar_control_agent segfaults, out-of-memory (OOM) errors, and TPU device initialization failures on TPU v6e nodes in GKE caused by race conditions during TPU device resets or high-frequency metrics polling. Use when troubleshooting vbar_control_agent crashes, memory cgroup OOMs in serial console logs, tpu-device-plugin metrics checksum corruption errors, or custom TPU metrics collection conflicts on GKE TPU v6e nodes. Don't use for general non-TPU container OOM troubleshooting or standard GKE node lifecycle operations.

55installs

Sourcegoogle/skills

Added on2026-08-05

NPX Install

npx skill4agent add google/skills gke-ai-troubleshooting-tpu-vbar-oom

SKILL.md Content

View Translation Comparison →

TPU Connection Failure and VBAR OOM Troubleshooting

Use this skill to systematically diagnose and prevent

vbar_control_agent

segfaults and Out-Of-Memory (OOM) errors on TPU v6e nodes.

⚠️ Prerequisites

Cloud Logging must be enabled for the project.
Access to the project and cluster via
```
gcloud
```
or equivalent tool.

🔍 Diagnostic Workflow

Step 0: Context Acquisition & Time Window Definition

Independently gather required context using available GCP/GKE tools or use the provided

{variable}

placeholders:

```
{project_id}
```
: The GCP Project ID (e.g.,
```
customer-ai-project-123
```
).
```
{cluster_name}
```
: The GKE Cluster Name (e.g.,
```
tpu-cluster-prod
```
).
```
{node_name}
```
: The Node Name or Instance ID (e.g.,
```
tpu-node-1
```
).
```
{workload_name}
```
: The Workload Name / JobSet Name (e.g.,
```
my-training-job-456
```
).
```
{namespace}
```
: The Workload Namespace.
```
{issue_time}
```
: The timestamp of the issue (e.g.,
```
2026-04-14T20:00:00Z
```
).

Time Handling & Execution Rules

Window Calculation: If an issue timestamp

{issue_time}

is provided, calculate the query time window as

[{issue_time} - 30m]

to

[{issue_time} + 30m]

.

Let
```
{start_time}
```
=
```
{issue_time} - 30m
```
Let
```
{end_time}
```
=
```
{issue_time} + 30m
```

Informational vs. Live Execution: If the user request is informational or query-formulation (e.g. "How can I check...", "How do I determine..."), or if live GCP project resources are not actively targetable, directly output the calculated time window, log names, and Cloud Logging filter templates without attempting live log execution commands.

Step 1: Check for

vbar_control_agent

OOMs

Look for specific

out of memory

messages from

vbar_control_agent

in serial console logs (

serialconsole.googleapis.com%2fserial_port_1_output

).

Tool to use:
```
query_logs
```
(for live diagnostics)
Filter Templates:

Serial Console Logs (OOMs):

sql

logName="projects/{project_id}/logs/serialconsole.googleapis.com%2fserial_port_1_output"
AND labels."compute.googleapis.com/resource_name"="{node_name}"
AND SEARCH(text_payload, "Memory cgroup out of memory: Killed process .* (vbar_control_ag)")
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

Logic: Presence of

Memory cgroup out of memory

messages related to

vbar_control_agent

. Stack traces pointing to

libtpu::tpunetd::VBARControlHelper::MetricsReadFromVBAR

are a strong indicator.

Automation: Proceed to next step automatically after reporting findings.
Reference: See
```
references/failure_signatures.md
```
for example log patterns.

Step 2: Investigate

tpu-device-plugin

Metrics Fetch Failures [Low Risk]

Check if

tpu-device-plugin

is reporting metric fetch failures.

Tool to use:
```
query_logs
```
Filter Template:

sql

resource.type="k8s_container"
AND resource.labels.project_id="{project_id}"
AND resource.labels.cluster_name="{cluster_name}"
AND resource.labels.container_name="tpu-device-plugin"
AND severity=ERROR
AND textPayload:"metrics fetch failed for .* deviceID and .* device path with error: checksum didn't match with the metrics data. Corrupt data found"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

Logic: Errors indicating "metrics fetch failed" with "checksum didn't match" suggest vBAR memory corruption.
Automation: Proceed to next step automatically after reporting findings.

Step 3: Check for Custom Metrics Collection Usage [Low Risk]

Inspect cluster configurations, workloads, or container specs to determine if custom TPU metrics collection mechanisms are deployed.

Action: Check if custom scripts or agents (e.g., using
```
libtpu.sdk.tpumonitoring
```
) are deployed that frequently query
```
GetHostMetrics
```
from
```
vBAR Control Agent
```
.

Verification Commands:

Kubectl Search (Inspect workload env/specs):

bash

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'

Log Search Filter (
query_logs
):

sql

resource.type="k8s_container"
AND resource.labels.project_id="{project_id}"
AND resource.labels.cluster_name="{cluster_name}"
AND textPayload:"libtpu.sdk.tpumonitoring"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

Logic: Confirmation of custom metrics collection helps confirm the race condition hypothesis.

🛠️ Resolution Workflow

Resolution 1: Temporarily Disable Custom Metrics Collection [High Risk]

If a custom metrics collection agent is identified, recommend disabling it.

Action: Recommend disabling the custom metrics collector.
Justification: Prevents reads from vBAR during device resets, stopping crashes and OOMs.

Resolution 2: Await

vbar_control_agent

Resiliency Update [Low Risk]

Advise that a permanent fix will be available in a future GKE version.

Action: Recommend upgrading GKE when the fix is available.
Justification: The updated agent will be resilient to memory corruption and gracefully handle reads from unbound vBARs.

📋 copypaste checklist

Acquire context and compute
```
[{start_time}, {end_time}]
```
window.
Check for
```
vbar_control_agent
```
segfaults and OOMs using
```
query_logs
```
.
Investigate
```
tpu-device-plugin
```
failures using
```
query_logs
```
.
Inspect for custom metrics collection usage.
Advise disabling custom metrics collection if applicable.
Advise awaiting resiliency update.