Back to Details

gke-node-notready

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

GKE Node NotReady Troubleshooting Skill

GKE Node NotReady 故障排查技能

Use this skill to systematically diagnose why one or more GKE nodes report a 
NotReady
(or 
Ready: Unknown
) status and to propose safe remediations. A 
NotReady
status means the node's kubelet is not reporting to the control plane correctly, so Kubernetes stops scheduling new Pods on the node, which can reduce application capacity and cause downtime.
This skill operates non-interactively and enforces a read-only diagnostics boundary: gather evidence first, then propose a fix (a 
kubectl
/
gcloud
command or a GitOps manifest change) for a human to apply. Never mutate the cluster, drain, delete, or recreate nodes automatically.
[!IMPORTANT] First rule out an expected 
NotReady
: a node that is newly provisioning, upgrading, being repaired, cordoned, or scaling down will transiently report 
NotReady
. Only treat it as a fault if it persists beyond the expected window.
使用本技能可系统性诊断一个或多个GKE节点报告
NotReady
（或
Ready: Unknown
）状态的原因，并提出安全的修复方案。
NotReady
状态意味着节点的kubelet未正确向控制平面上报状态，因此Kubernetes会停止在该节点上调度新Pod，这可能会降低应用容量并导致停机。
本技能以非交互方式运行，并严格遵循只读诊断边界：先收集证据，再提出修复方案（
kubectl
/
gcloud
命令或GitOps清单变更）供人工执行。绝对不要自动修改集群、驱逐、删除或重建节点。
[!IMPORTANT] 首先排除预期内的
NotReady
状态：新配置、正在升级、修复、封锁或缩容的节点会暂时报告
NotReady
。只有当该状态持续超出预期时间窗口时，才将其视为故障。

🔍 Diagnostic Workflow

🔍 诊断流程

Step 0: Context discovery & time window

步骤0：上下文发现与时间窗口

  1. Parameter extraction — obtain 
    project_id
    , 
    cluster_name
    , 
    cluster_location
    , and 
    node_name
    non-interactively from the user prompt, active 
    SETTINGS.md
    , or environment defaults (
    kubectl config current-context
    , 
    gcloud config get-value project
    ).
  2. Credentials & fallback — attempt 
    gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id}
    . If the cluster is unreachable or commands fail (sandbox/dry-run/offline), present the exact diagnostic commands for a human to run and continue the analysis from the reported symptoms.
  3. Time window — determine 
    {issue_time}
    (explicit, relative, or now) and center a 1-hour window around it (
    start = issue_time - 30m
    , 
    end = issue_time + 30m
    ) for all log/metric queries.
  1. 参数提取 — 从用户提示、活跃的
    SETTINGS.md
    或环境默认值（
    kubectl config current-context
    、
    gcloud config get-value project
    ）中自动获取
    project_id
    、
    cluster_name
    、
    cluster_location
    和
    node_name
    。
  2. 凭证与回退方案 — 尝试执行
    gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id}
    。如果集群无法访问或命令执行失败（沙箱/试运行/离线环境），则提供供人工运行的准确诊断命令，并根据报告的症状继续分析。
  3. 时间窗口 — 确定
    {issue_time}
    （明确指定、相对时间或当前时间），并围绕该时间设置1小时的窗口（
    start = issue_time - 30m
    ，
    end = issue_time + 30m
    ），用于所有日志/指标查询。

Step 1: Identify NotReady nodes and gather initial status

步骤1：识别NotReady节点并收集初始状态

bash
undefined
bash
undefined

List nodes and spot NotReady status, node IPs, and container-runtime version.

列出节点并查看NotReady状态、节点IP和容器运行时版本。

kubectl get nodes -o wide
kubectl get nodes -o wide

Inspect the affected node's Conditions and Events (the primary clues).

检查受影响节点的状态条件和事件（主要线索）。

kubectl describe node "{node_name}"

Equivalent via Cloud Logging (preferred when kubectl access is limited or for
historical events). Open it as a **Logs Explorer deep link** — URL-encode the
query and append the project and Step 0 time window:
`https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id}`
(encode `/` as `%2F`, or use `;duration=PT1H` for a rolling hour):
resource.type="k8s_node" log_id("events") resource.labels.node_name="{node_name}" resource.labels.cluster_name="{cluster_name}" resource.labels.location="{cluster_location}"

**Interpret the `Conditions` table:**

- `Ready: False` / `Ready: Unknown` with reason `KubeletNotReady` /
  `NodeStatusUnknown` ("Kubelet stopped posting node status") → kubelet or
  runtime problem; continue to Step 2.
- `MemoryPressure: True`, `DiskPressure: True`, `PIDPressure: True` → resource
  exhaustion; go to Step 4b.
- `NetworkUnavailable: True` → networking/CNI problem; go to Step 4d.

--------------------------------------------------------------------------------
kubectl describe node "{node_name}"

当kubectl访问受限或需要查询历史事件时，优先使用Cloud Logging的等效方式。将其作为**Logs Explorer深度链接**打开——对查询进行URL编码，并附加项目信息和步骤0的时间窗口：
`https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id}`
（将`/`编码为`%2F`，或使用`;duration=PT1H`表示最近1小时）：
resource.type="k8s_node" log_id("events") resource.labels.node_name="{node_name}" resource.labels.cluster_name="{cluster_name}" resource.labels.location="{cluster_location}"

**解读`Conditions`表格：**

- `Ready: False` / `Ready: Unknown`，原因是`KubeletNotReady` / `NodeStatusUnknown`（"Kubelet停止上报节点状态"）→ kubelet或运行时问题；继续执行步骤2。
- `MemoryPressure: True`、`DiskPressure: True`、`PIDPressure: True` → 资源耗尽；跳至步骤4b。
- `NetworkUnavailable: True` → 网络/CNI问题；跳至步骤4d。

--------------------------------------------------------------------------------

Step 2: Scan kubelet logs for error signatures

步骤2：扫描kubelet日志查找错误特征

Open these kubelet logs as a Logs Explorer deep link using the same 
logs/query;query={URL_ENCODED_QUERY};timeRange=...?project=...
pattern as Step 1.
resource.type="k8s_node"
resource.labels.node_name="{node_name}"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
log_id("kubelet")
severity>=WARNING
Also review the node's serial-console logs (
log_id("serialconsole.googleapis.com/serial_port_1_output")
or the 
resource.type="gce_instance"
serial logs) for kernel 
TaskHung
, OOM-killer, or disk I/O errors that correlate with the kubelet failures.
使用与步骤1相同的
logs/query;query={URL_ENCODED_QUERY};timeRange=...?project=...
格式，将这些kubelet日志作为Logs Explorer深度链接打开。
resource.type="k8s_node"
resource.labels.node_name="{node_name}"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
log_id("kubelet")
severity>=WARNING
同时查看节点的串行控制台日志（
log_id("serialconsole.googleapis.com/serial_port_1_output")
或
resource.type="gce_instance"
的串行日志），查找与kubelet故障相关的内核
TaskHung
、OOM-killer或磁盘I/O错误。

Step 3: Map the signature to a root cause (decision table)

步骤3：将特征映射到根本原因（决策表）

Kubelet / event signatureLikely root causeGo to
runtime is down
, 
Container runtime not ready
, errors on 
/run/containerd/containerd.sock
(connection refused / DeadlineExceeded)		Container runtime (
containerd
) down or unresponsive		Step 4a
Got sys oom event from cadvisor
/ kernel OOM-killer in serial logs		System (node-level) OOM killed critical processesStep 4b
PLEG is not healthy
PLEG stalled, usually node overload (CPU/disk)Step 4c
TaskHung
for 
containerd
/
kubelet
, high disk latency		Disk throttling / I/O starvationStep 4b
failed to ensure lease
, 
leases.coordination.k8s.io ... namespace kube-node-lease ... terminating
kube-node-lease
termination → NotReady flapping		Step 4f
Kubelet cannot reach API server, TLS/dial timeoutsKubelet ↔ control-plane connectivityStep 4d
NetworkPluginNotReady
, 
cni plugin not initialized
, 
NetworkUnavailable
CNI plugin failureStep 4d
Node-critical DaemonSet Pods (CNI, kube-proxy, metadata) blocked from admissionAdmission webhook interferenceStep 4e
Only generic 
NodeNotReady
, no other signature		Cause unclear — widen to Step 4d, then escalateEscalation
Kubelet / 事件特征可能的根本原因跳转至
runtime is down
、
Container runtime not ready
、
/run/containerd/containerd.sock
相关错误（连接拒绝/DeadlineExceeded）		容器运行时（
containerd
）停止或无响应		步骤4a
Got sys oom event from cadvisor
/ 串行日志中的内核OOM-killer		系统（节点级）OOM杀死了关键进程步骤4b
PLEG is not healthy
PLEG停滞，通常是节点过载（CPU/磁盘）步骤4c
containerd
/
kubelet
出现
TaskHung
、磁盘延迟高		磁盘限流 / I/O资源耗尽步骤4b
failed to ensure lease
、
leases.coordination.k8s.io ... namespace kube-node-lease ... terminating
kube-node-lease
终止 → NotReady状态波动		步骤4f
Kubelet无法连接API服务器、TLS/拨号超时Kubelet ↔ 控制平面连接问题步骤4d
NetworkPluginNotReady
、
cni plugin not initialized
、
NetworkUnavailable
CNI插件故障步骤4d
节点关键DaemonSet Pod（CNI、kube-proxy、元数据）被准入控制拦截准入Webhook干扰步骤4e
仅出现通用的
NodeNotReady
，无其他特征		原因不明 — 扩展至步骤4d，然后升级处理升级处理

Step 4: Branch investigations

步骤4：分支调查

4a. Container runtime (
containerd
) down

4a. 容器运行时（
containerd
）停止

Confirm the kubelet cannot talk to containerd (socket errors above). Check for 
containerd
restarts/crashes in serial logs. Remediation (propose, don't run): recreate/repair the node (
kubectl drain
then let the node pool recreate it, or 
gcloud container clusters upgrade
/node auto-repair); if it recurs across nodes, suspect a node image or custom DaemonSet interfering with containerd.
确认kubelet无法与containerd通信（如上述套接字错误）。在串行日志中检查
containerd
是否重启/崩溃。修复方案（仅建议，请勿执行）： 重建/修复节点（
kubectl drain
后让节点池自动重建，或执行
gcloud container clusters upgrade
/节点自动修复）；如果多个节点出现此问题，怀疑节点镜像或自定义DaemonSet干扰了containerd。

4b. Resource pressure & OOM

4b. 资源压力与OOM

bash
undefined
bash
undefined

Node allocatable vs. usage.

节点可分配资源与使用情况。

kubectl describe node "{node_name}" | sed -n '/Allocated resources/,/Events/p'
Cloud Monitoring metrics to inspect (read-only): `kubernetes.io/node/memory/used_bytes`,
`kubernetes.io/node/cpu/core_usage_time`, `kubernetes.io/node/ephemeral_storage/used_bytes`.
- **DiskPressure / disk throttling**: full boot disk or slow PD → increase disk
  size / use a faster PD type; reduce image/log churn.
- **System OOM**: node memory exhausted → set/raise Pod memory `requests`/`limits`,
  reduce over-commit, or use larger machine types. Distinguish **system OOM**
  (node-wide, kills kubelet/runtime) from **cgroup OOM** (single container).
- **PIDPressure**: too many processes → cap Pod PIDs / reduce workload density.
kubectl describe node "{node_name}" | sed -n '/Allocated resources/,/Events/p'
需要检查的Cloud监控指标（只读）：`kubernetes.io/node/memory/used_bytes`、`kubernetes.io/node/cpu/core_usage_time`、`kubernetes.io/node/ephemeral_storage/used_bytes`。
- **DiskPressure / 磁盘限流**：启动盘已满或PD速度慢 → 增大磁盘容量 / 使用更快的PD类型；减少镜像/日志生成量。
- **系统OOM**：节点内存耗尽 → 设置/提高Pod内存`requests`/`limits`，减少超配，或使用更大规格的机器类型。区分**系统OOM**（节点级，杀死kubelet/运行时）与**cgroup OOM**（单个容器）。
- **PIDPressure**：进程过多 → 限制Pod的PID数量 / 降低工作负载密度。

4c. PLEG is not healthy

4c. PLEG状态异常

PLEG is not healthy
almost always means the node is overloaded (CPU saturation, disk latency, or too many Pods/containers per node) so the runtime can't relist in time. Correlate with 4b metrics. Remediation: reduce node density, add CPU/disk headroom, or spread workloads.
PLEG is not healthy
几乎总是意味着节点过载（CPU饱和、磁盘延迟高或每个节点的Pod/容器数量过多），导致运行时无法及时重新列出资源。结合步骤4b的指标进行关联分析。**修复方案：**降低节点密度，增加CPU/磁盘余量，或分散工作负载。

4d. Networking

4d. 网络问题

bash
undefined
bash
undefined

Are node-critical networking Pods healthy on this node?

该节点上的节点关键网络Pod是否健康？

kubectl get pods -n kube-system -o wide --field-selector spec.nodeName={node_name}
- **Kubelet ↔ control-plane**: dial/TLS timeouts to the API server → check
  firewall rules, Private Google Access, authorized networks, and route/NAT
  changes.
- **CNI failure** (`NetworkPluginNotReady`): the CNI DaemonSet
  (`netd`/`calico`/dataplane) is not running on the node → inspect those Pods'
  logs/events.
kubectl get pods -n kube-system -o wide --field-selector spec.nodeName={node_name}
- **Kubelet ↔ 控制平面**：连接API服务器时出现拨号/TLS超时 → 检查防火墙规则、Private Google Access、授权网络以及路由/NAT变更。
- **CNI故障**（`NetworkPluginNotReady`）：CNI DaemonSet（`netd`/`calico`/数据平面）未在节点上运行 → 检查这些Pod的日志/事件。

4e. Admission webhook interference

4e. 准入Webhook干扰

A misconfigured/failing validating or mutating webhook with a broad scope can block node-critical system Pods from being admitted, keeping the node NotReady.
bash
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
Look for webhooks that intercept 
kube-system
/ node-critical objects with 
failurePolicy: Fail
. Remediation (propose): scope the webhook out of 
kube-system
/node-critical namespaces or set an appropriate 
namespaceSelector
.
配置错误或故障的验证/变更Webhook如果范围过广，可能会阻止节点关键系统Pod被准入，导致节点处于NotReady状态。
bash
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
查找拦截
kube-system
/ 节点关键对象且
failurePolicy: Fail
的Webhook。**修复方案（建议）：**将Webhook范围排除在
kube-system
/节点关键命名空间之外，或设置合适的
namespaceSelector
。

4f. 
kube-node-lease
termination flapping

4f. 
kube-node-lease
终止波动

If the node flaps NotReady with 
leases.coordination.k8s.io ... namespace kube-node-lease ... is being terminated
, the 
kube-node-lease
namespace was deleted/terminating. Remediation (propose): do not delete the 
kube-node-lease
namespace; if terminating, identify the finalizer/actor holding it and restore the namespace.
如果节点的NotReady状态反复出现，且伴随
leases.coordination.k8s.io ... namespace kube-node-lease ... is being terminated
信息，说明
kube-node-lease
命名空间已被删除/处于终止状态。**修复方案（建议）：**不要删除
kube-node-lease
命名空间；如果处于终止状态，找出持有它的终结器/执行者并恢复该命名空间。

Step 5: Remediation boundary & escalation

步骤5：修复边界与升级处理

  • Present the root cause + evidence (the exact conditions, events, log lines, or metrics observed). Provide Cloud Logging deep links (and Cloud Monitoring links for the Step 4b metrics) to the supporting entries — using the deep-link pattern from Steps 1-2 — so a human can open the evidence directly.
  • Propose the fix as a command or GitOps manifest change for a human to apply — never apply, drain, or recreate nodes automatically. When to escalate (do this instead of proposing more self-service diagnostics):
Escalate when either:
  • the relevant logs are unavailable — excluded by a logging filter, or older than the log bucket's retention (the 
    _Default
    bucket defaults to 30 days, so incidents older than that are permanently deleted); or
  • the kubelet/event signature is not in the Step 3 table and the root cause remains undetermined after the branch investigations.
In those cases, do all three:
  1. State the limitation plainly (for example, "kubelet logs for that date are past the 30-day 
    _Default
    retention window and are permanently deleted").
  2. Summarize the findings you did gather (node conditions, events, metrics, and any Admin Activity audit logs still in the 
    _Required
    bucket, default 400-day retention).
  3. Route to GKE support / engineering escalation with those findings. Do not keep proposing further self-service investigation, and do not fabricate a diagnosis when the evidence is missing.
  • 呈现根本原因 + 证据（观察到的具体状态条件、事件、日志行或指标）。提供Cloud Logging深度链接（以及步骤4b指标的Cloud监控链接）指向相关记录——使用步骤1-2中的深度链接格式，以便人工直接查看证据。
  • 将修复方案作为命令或GitOps清单变更提出，供人工执行 — 绝对不要自动执行、驱逐或重建节点。 何时升级处理（此时应停止自助诊断，转而执行以下操作）：
出现以下任一情况时升级处理：
  • 相关日志不可用 — 被日志过滤器排除，或早于日志存储桶的保留期限（
    _Default
    存储桶默认保留30天，因此超过该期限的事件会被永久删除）；或
  • kubelet/事件特征不在步骤3的表格中，且经过分支调查后根本原因仍无法确定。
在这些情况下，需执行以下三项操作：
  1. 明确说明限制（例如："该日期的kubelet日志已超过
    _Default
    存储桶30天的保留期限，已被永久删除"）。
  2. 总结已收集的发现（节点状态条件、事件、指标，以及
    _Required
    存储桶中仍保留的任何管理员活动审计日志，默认保留400天）。
  3. 将这些发现提交给GKE支持团队/工程升级处理。请勿继续提出更多自助调查建议，且在证据缺失时请勿编造诊断结果。

References

参考资料

This skill is derived from public Google Cloud documentation:
本技能源自Google Cloud公开文档：
  • 排查NotReady状态的节点 — 节点状态条件以及kubelet / PLEG / 系统OOM / containerd / 
    kube-node-lease
    / CNI / 准入Webhook的特征及其修复方案。
  • 排查节点注册问题 — 用于排查从未完成注册的节点的Node Registration Checker工具。
  • 查看GKE日志 和 Cloud Logging路由概述 — 日志查询（
    resource.type="k8s_node"
    、
    log_id("kubelet")
    ）和日志存储桶保留期限（
    _Default
    为30天，
    _Required
    为400天）。
  • Logs Explorer界面 — 通过URL构建和共享查询（上述使用的
    logs/query;query=...
    深度链接格式）。