project

cluster_name

location

namespace

default

issue_time

[issue_time - 30m, issue_time + 30m]

--start_time="{start_time}" --duration=1h

{start_time}

issue_time - 30m

2026-10-22T11:10:00Z

--limit

• If the window contains more than one failure, report each with timestamps and say which one matches the user's symptom.
• Report the root cause and the workaround separately. The root cause is the component that logged the earliest stall or error (for example the proxy, the RM or a worker). A configuration change that avoids that code path (a different checkpoint handler, flag or topology) is a workaround or a contributing factor: list it under the fix, not as the root cause.
• When you compare with other runs (an earlier attempt, a sibling test that passed, or the same test on other days), use the comparison to narrow down which code path is affected, not as proof of the cause. A setting that differs in a passing run does not explain the failure if a Pathways component logged the stall or error.
• End the report with the checks you ran and their result, so the user can repeat them: preflight, the actions or

  gcloud logging read

  queries you used (with their time window), pod events, and node OOMs. Include checks that failed or could not run (e.g. no log access) and say what to run instead.

[!TIP]

Always check the recent chat context or shared log queries from the user. Often, all five fields can be directly scraped from their console links or queries!

Suspended

• Potential Root Causes:
  • The JobSet name is too long. Because JobSet appends replicated job names and index numbers to child resources, long prefixes fail standard Kubernetes resource name validation (max 63/253 chars).
  • The JobSet YAML is malformed (e.g. invalid or missing fields, wrong resource requests or node selectors), so the workload is never admitted to the queue and stays suspended. Ask the user for the YAML (see Retrieving Active YAML Configuration) and check it for errors.
• Diagnosis: Check the JobSet controller manager logs in

  jobset-system

  namespace to find truncation, length limit, name-generation, or validation errors. You can use the debug_workload script to quickly query the logs:

python3 scripts/debug_workload.py \
  --project="{project}" \
  --cluster_name="{cluster_name}" \
  --location="{location}" \
  --jobset_name="{jobset_name}" \
  --start_time="{start_time}" --duration=1h \
  --action=summary \
  --execute

• SIGTERMs are the most common mechanism for GKE to stop/restart workloads. A SIGTERM is a symptom; find out who sent it.
• Common senders: Spot VM preemption, Compute Engine host maintenance, host errors, node auto-repair, or a JobSet restart triggered by another failed container. Check infrastructure interruptions and pod events for the time window:
  bash

  python3 scripts/debug_workload.py \
    --project="{project}" \
    --cluster_name="{cluster_name}" \
    --location="{location}" \
    --jobset_name="{jobset_name}" \
    --start_time="{start_time}" --duration=1h \
    --action=interruptions \
    --execute
  # then the same command with --action=pod-events

• If no infrastructure event explains the SIGTERM, trace the failure back from the user workload to the TPU that failed first:
  1. In the user workload logs (

     --action=user-workload

     ), find the time of the first error or the last completed step.
  2. In the RM logs (

     --action=pathways-rm

     ) around that time, find when the first failure occurred or the first TPU slice/worker stopped responding (e.g. a slice leaving the ready state or

     Job cancelled due to ...

     ), and which slice/worker it was.
  3. Inspect that worker's logs (

     --action=workers

     , adding

     --min_severity=WARNING

     if needed) and its node's events (

     --action=pod-events

     ,

     --action=node-ooms

     ,

     --action=interruptions

     ) to find out what happened there.
  4. The cause can be a user code error (unlikely) or another fault, such as a hardware failure, an OOM, a crash in the worker, or a node interruption.

• kubectl get events -n {namespace} --field-selector involvedObject.name={jobset_name}

  only works for live incidents: Kubernetes keeps events for a certain time only. For past incidents use

  --action=pod-events

  /

  --action=summary

  , which read the events from Cloud Logging.
• For deeper analysis of restarts and interruptions, follow the public gke-ai-troubleshooting-jobset-interruption skill; for host maintenance, follow gke-ai-troubleshooting-handle-disruption-gpu-tpu.

debug_workload.py

user-workload

main

jax-tpu

--action=user-workload

--action=proxy

Connection refused

:29001

Socket closed

pathways-rm

--action=pathways-rm

Build label

pathways-worker

--action=workers

HANG_DETECTED

--action=colocated-python-sidecar

• A client-side

  Unable to initialize backend 'proxy': UNAVAILABLE: Unable to establish connection

  is a symptom. Check whether the proxy, RM and workers started (see Parsing Important Logs) and find their earliest error before you suspect the user's code.

• IFRT proxy server was terminated: UNAVAILABLE: Socket closed

  and

  SLICE_FAILURE_CHIP_DRIVER_ERROR

  are downstream errors: go to the RM and worker logs.
• When one worker fails, the other workers of the slice fail shortly after. The root cause is in the logs of the first worker that failed.
• The built-in search terms only cover known failures. If an action returns nothing useful, re-run it with

  --min_severity=WARNING

  (

  DEFAULT

  for the user workload) and add the workload's own strings with

  --extra_terms

  .
• If the logs still don't explain the failure, read all logs of the container (F), then ask for more verbose proxy logs or XProfs (G).

HANG_DETECTED

• Always use double quotes for strings, uppercase

  AND

  /

  OR

  /

  NOT

  , and parentheses to group terms.
• Restrict

  resource.type

  (

  k8s_container

  ,

  k8s_pod

  ,

  k8s_node

  ,

  gke_nodepool

  ,

  gce_instance

  ) and use

  log_id("events")

  ,

  log_id("stdout")

  or

  log_id("stderr")

  where possible.
• Always add a time window (

  timestamp>="..."

  and

  timestamp<="..."

  ).
• Pathways components log glog lines into

  jsonPayload.message

  (some containers) or

  textPayload

  (others); bare quoted terms or

  SEARCH()

  match both.
• Pod label keys are stored with

  .

  replaced by

  _

  , e.g.

  labels."k8s-pod/batch_kubernetes_io/job-completion-index"

  and

  labels."k8s-pod/jobset_sigs_k8s_io/jobset-name"

  . Some public examples spell the Job label with dots (

  batch.kubernetes.io/job-name

  ); that form does not match Pathways logs.