huawei-cloud-cce-metric-analyzer

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Huawei Cloud CCE Metric Analyzer

华为云CCE指标分析器

Overview

概述

Query and analyze metrics for CCE clusters (Pod/Node CPU/memory/disk) and cloud resources (ECS, ELB, EIP, NAT). Supports threshold-based anomaly detection, status classification (critical/warning/normal), and full-cluster monitoring aggregation.

Architecture:

python3 scripts/huawei-cloud.py

dispatcher → hcloud (KooCLI) cloud service queries + signed AOM Prometheus HTTP queries + limited kubectl reads only when Kubernetes resource relationships are required → Pod/Node metrics, ECS/ELB/EIP/NAT metrics → Threshold classification → Anomaly detection

Execution method: Cloud service queries are executed through the local
hcloud
CLI. AOM Prometheus
query_range
calls are the only exception and use signed HTTPS requests because the required Prometheus range-query path is not compatible with hcloud. Do not call Huawei Cloud SDKs, curl IAM flows, openstack, or hand-written cloud APIs outside the bundled dispatcher.

Related Skills: use pod/node diagnosers, Kubernetes event analyzer, capacity/cost skills, or auto-remediation runner for follow-up diagnosis or explicitly requested remediation.

Capabilities:

Pod CPU/memory TopN ranking and single Pod time-series metrics
Node CPU/memory/disk TopN ranking and single Node time-series metrics
Node GPU and xGPU metrics, including GPU utilization, memory, temperature, power, schedule policy, xGPU allocation, usage, and health
CoreDNS QPS, error rate excluding NXDOMAIN, NXDOMAIN rate, P95 latency, replica count, and per-Pod CPU/memory metrics
nginx-ingress QPS, 4xx/5xx rate, success rate, P95 latency, active connections, per-Pod CPU/memory, and Ingress TLS certificate expiration status
Autoscaler unschedulable Pods, node state count, scale-up/down events, errors, node groups, HPA current/desired replicas, and per-Pod CPU/memory metrics
Kubernetes control-plane metrics for apiserver, etcd, controller-manager, and scheduler
ECS instance CPU/memory/disk/network metrics
ELB connection, bandwidth, QPS metrics
EIP bandwidth, traffic, packet loss metrics
NAT Gateway SNAT connection metrics
Full-cluster monitoring aggregation with anomaly detection (80% threshold)
Threshold-based status classification (critical/warning/normal/unknown)

Typical Use Cases: query Pod/Node TopN, GPU/xGPU, CoreDNS, nginx-ingress, autoscaler, control-plane, ECS/ELB/EIP/NAT metrics, full-cluster aggregation, and threshold-based anomaly detection.

查询并分析CCE集群（Pod/Node的CPU/内存/磁盘）及云资源（ECS、ELB、EIP、NAT）的指标。支持基于阈值的异常检测、状态分类（严重/警告/正常）以及全集群监控聚合。

架构：

python3 scripts/huawei-cloud.py

调度器 → hcloud（KooCLI）云服务查询 + 签名的AOM Prometheus HTTP查询 + 仅在需要Kubernetes资源关系时进行有限的kubectl读取 → Pod/Node指标、ECS/ELB/EIP/NAT指标 → 阈值分类 → 异常检测

执行方式：云服务查询通过本地
hcloud
CLI执行。AOM Prometheus的
query_range
调用是唯一例外，使用签名HTTPS请求，因为所需的Prometheus范围查询路径与hcloud不兼容。请勿在捆绑的调度器之外调用华为云SDK、curl IAM流程、openstack或手写云API。

相关技能：使用Pod/Node诊断器、Kubernetes事件分析器、容量/成本技能或自动修复运行器进行后续诊断或用户明确要求的修复操作。

功能:

Pod CPU/内存TopN排名及单个Pod的时间序列指标
Node CPU/内存/磁盘TopN排名及单个Node的时间序列指标
Node GPU和xGPU指标，包括GPU利用率、内存、温度、功耗、调度策略、xGPU分配、使用情况及健康状态
CoreDNS的QPS、排除NXDOMAIN的错误率、NXDOMAIN率、P95延迟、副本数及每个Pod的CPU/内存指标
nginx-ingress的QPS、4xx/5xx错误率、成功率、P95延迟、活跃连接数、每个Pod的CPU/内存，以及Ingress TLS证书过期状态
Autoscaler的不可调度Pod数、节点状态计数、扩缩容事件、错误信息、节点组、HPA当前/期望副本数及每个Pod的CPU/内存指标
Kubernetes控制面的apiserver、etcd、controller-manager和scheduler指标
ECS实例的CPU/内存/磁盘/网络指标
ELB的连接数、带宽、QPS指标
EIP的带宽、流量、丢包率指标
NAT网关的SNAT连接数指标
带异常检测（80%阈值）的全集群监控聚合
基于阈值的状态分类（严重/警告/正常/未知）

典型用例：查询Pod/Node TopN、GPU/xGPU、CoreDNS、nginx-ingress、autoscaler、控制面、ECS/ELB/EIP/NAT指标，全集群聚合，以及基于阈值的异常检测。

Prerequisites

前提条件

1. Runtime Dependencies

1. 运行时依赖

Python 3.8+ for the dispatcher and result processing
hcloud (KooCLI) 7.2.2+ for CCE/ECS/ELB/VPC/EIP/NAT/CES/IAM cloud service queries
```
kubectl
```
only for Kubernetes resource reads that cannot be derived from AOM/hcloud, such as Pod
```
label_selector
```
filtering, Ingress TLS certificate checks, and LoadBalancer Service discovery for ELB/EIP association; clusters without external EIP require the
```
kubectl cce
```
plugin from
```
kubectl-cce-plugin/README.md
```
Prometheus-related monitoring data is queried from AOM Prometheus with signed HTTPS requests; the cluster must have the Prometheus add-on integrated with AOM, otherwise these tools may return empty metric series
Controller-manager, scheduler, and etcd metrics require the
```
kube-controller-manager
```
,
```
kube-scheduler
```
, and
```
etcd-server
```
ServiceMonitors to be enabled separately in AOM; otherwise these tools may return empty metric series
Autoscaler, ingress-controller, and NVIDIA GPU metrics require the corresponding
```
autoscaler
```
,
```
ingress-controller
```
, and
```
nvidia-gpu-device-plugin
```
PodMonitors to be enabled separately in AOM; ingress request metrics also require
```
nginx_ingress_controller_requests
```
to be explicitly allowed in the ingress-controller PodMonitor
Run environment check before first use (see Verification section)
Install and use
```
kubectl-cce
```
according to references/kubectl-cce.md

调度器和结果处理需要Python 3.8+
CCE/ECS/ELB/VPC/EIP/NAT/CES/IAM云服务查询需要hcloud（KooCLI）7.2.2+
```
kubectl
```
仅用于无法从AOM/hcloud获取的Kubernetes资源读取，例如Pod
```
label_selector
```
过滤、Ingress TLS证书检查，以及用于ELB/EIP关联的LoadBalancer Service发现；无外部EIP的集群需要
```
kubectl-cce-plugin/README.md
```
中的
```
kubectl cce
```
插件
Prometheus相关监控数据通过签名HTTPS请求从AOM Prometheus查询；集群必须集成Prometheus插件与AOM，否则这些工具可能返回空指标序列
Controller-manager、scheduler和etcd指标需要在AOM中单独启用
```
kube-controller-manager
```
、
```
kube-scheduler
```
和
```
etcd-server
```
的ServiceMonitors；否则这些工具可能返回空指标序列
Autoscaler、ingress-controller和NVIDIA GPU指标需要在AOM中单独启用对应的
```
autoscaler
```
、
```
ingress-controller
```
和
```
nvidia-gpu-device-plugin
```
的PodMonitors；ingress请求指标还需要在ingress-controller的PodMonitor中明确允许
```
nginx_ingress_controller_requests
```
首次使用前运行环境检查（见验证部分）
根据references/kubectl-cce.md安装并使用
```
kubectl-cce
```

2. Credential Configuration

2. 凭证配置

Valid Huawei Cloud credentials via hcloud profile or AK/SK mode
Security Rules:
- 🚫 Never expose AK/SK values in code, conversation, or commands
- 🚫 Never use
```
echo $HUAWEI_AK
```
  or
```
echo $HUAWEI_SK
```
  to check credentials
- ✅ Credential priority for hcloud calls is: explicit tool parameters > local hcloud profile > environment variables
- ✅ AOM Prometheus signed HTTP and Kubernetes certificate setup cannot use encrypted hcloud profile material, so they use explicit tool parameters first and environment variables as the signing fallback
- ✅ Prefer IAM users over root account for cloud operations
- ✅ Enable MFA for sensitive operations

Configuration Method:

bash

hcloud configure list

export HUAWEI_AK=<your-ak>
export HUAWEI_SK=<your-sk>
export HUAWEI_REGION=cn-north-4

通过hcloud配置文件或AK/SK模式提供有效的华为云凭证
安全规则:
- 🚫 切勿在代码、对话或命令中暴露AK/SK值
- 🚫 切勿使用
```
echo $HUAWEI_AK
```
  或
```
echo $HUAWEI_SK
```
  检查凭证
- ✅ hcloud调用的凭证优先级：显式工具参数 > 本地hcloud配置文件 > 环境变量
- ✅ AOM Prometheus签名HTTP和Kubernetes证书设置无法使用加密的hcloud配置文件内容，因此优先使用显式工具参数，环境变量作为签名的备选方案
- ✅ 云操作优先使用IAM用户而非根账户
- ✅ 敏感操作启用MFA

配置方法:

bash

hcloud configure list

export HUAWEI_AK=<your-ak>
export HUAWEI_SK=<your-sk>
export HUAWEI_REGION=cn-north-4

3. IAM Permission Requirements

3. IAM权限要求

API Action	Permission	Purpose
`cce:cluster:get`	Get cluster	View CCE cluster details
`aom:instance:list`	List AOM instances	Discover AOM Prometheus instance for metrics
`aom:metricsData:get`	Get metrics data	Query Pod/Node CPU/memory/disk metrics
`ces:metricsData:get`	Get CES metrics	Query ECS/ELB/EIP/NAT cloud resource metrics
`ecs:cloudServers:list`	List ECS servers	Correlate ECS instance IDs
`elb:loadbalancers:list`	List ELB instances	Correlate ELB IDs
`vpc:eips:list`	List EIPs	Correlate EIP IDs
`nat:natGateways:list`	List NAT Gateways	Correlate NAT Gateway IDs

Permission Failure Handling:

When any command fails due to IAM permission errors, display the required permission list
Guide the user to create a custom policy in the IAM console and grant authorization
Pause execution and wait for user confirmation that permissions have been granted

API操作	权限	用途
`cce:cluster:get`	获取集群	查看CCE集群详情
`aom:instance:list`	列出AOM实例	发现用于指标查询的AOM Prometheus实例
`aom:metricsData:get`	获取指标数据	查询Pod/Node的CPU/内存/磁盘指标
`ces:metricsData:get`	获取CES指标	查询ECS/ELB/EIP/NAT云资源指标
`ecs:cloudServers:list`	列出ECS服务器	关联ECS实例ID
`elb:loadbalancers:list`	列出ELB实例	关联ELB ID
`vpc:eips:list`	列出EIPs	关联EIP ID
`nat:natGateways:list`	列出NAT网关	关联NAT网关ID

权限失败处理:

当任何命令因IAM权限错误失败时，显示所需权限列表
引导用户在IAM控制台创建自定义策略并授予权限
暂停执行，等待用户确认权限已授予

Core Commands

核心命令

All commands use the Python dispatcher script:

python3 scripts/huawei-cloud.py <action> <key=value>...

所有命令均使用Python调度器脚本：

python3 scripts/huawei-cloud.py <action> <key=value>...

KooCLI命令格式标准

Do not ask users to run raw

hcloud

commands directly. Use the dispatcher format:

bash

python3 scripts/huawei-cloud.py <tool-name> key=value key=value

The dispatcher converts cloud service queries to KooCLI calls. AOM Prometheus range queries use signed HTTPS requests because that path is not compatible with hcloud. Avoid Kubernetes resource reads unless the tool explicitly needs Pod labels, Ingress TLS Secrets, or LoadBalancer Services. Quote values containing spaces,

, JSON, or PromQL; never print or persist AK/SK, security tokens, kubeconfig files, or temporary payloads; keep Kubernetes/AOM PromQL scoped with

cluster="<cluster_id>"

请勿要求用户直接运行原始

hcloud

命令。使用调度器格式：

bash

python3 scripts/huawei-cloud.py <tool-name> key=value key=value

调度器会将云服务查询转换为KooCLI调用。AOM Prometheus范围查询使用签名HTTPS请求，因为该路径与hcloud不兼容。除非工具明确需要Pod标签、Ingress TLS Secrets或LoadBalancer Services，否则避免读取Kubernetes资源。对包含空格、

、

、JSON或PromQL的值添加引号；切勿打印或持久化AK/SK、安全令牌、kubeconfig文件或临时负载；使用

cluster="<cluster_id>"

限定Kubernetes/AOM PromQL的范围。

1. CCE Pod Metrics

1. CCE Pod指标

bash

undefined

bash

undefined

Pod TopN — cluster-wide CPU/memory ranking

Pod TopN — 集群范围的CPU/内存排名

python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics_topN
region=cn-north-4 cluster_id=<cluster-id>
namespace=default top_n=10 hours=1

Pod TopN with label selector

带标签选择器的Pod TopN

python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics_topN
region=cn-north-4 cluster_id=<cluster-id>
namespace=default label_selector="app=nginx,version=v1" top_n=10 hours=1

Single Pod time-series

单个Pod的时间序列

python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics
region=cn-north-4 cluster_id=<cluster-id>
pod_name=my-app-xxx namespace=default hours=1

Single Pod GPU and xGPU metrics

单个Pod的GPU和xGPU指标

python3 scripts/huawei-cloud.py huawei_get_cce_pod_gpu_metrics
region=cn-north-4 cluster_id=<cluster-id>
pod_name=my-gpu-app-xxx namespace=default hours=1

undefined

python3 scripts/huawei-cloud.py huawei_get_cce_pod_gpu_metrics
region=cn-north-4 cluster_id=<cluster-id>
pod_name=my-gpu-app-xxx namespace=default hours=1

undefined

2. CCE Node Metrics

2. CCE Node指标

bash

undefined

bash

undefined

Node TopN — cluster-wide CPU/memory/disk ranking

Node TopN — 集群范围的CPU/内存/磁盘排名

python3 scripts/huawei-cloud.py huawei_get_cce_node_metrics_topN
region=cn-north-4 cluster_id=<cluster-id>
top_n=10 hours=1

Single Node time-series

单个Node的时间序列

python3 scripts/huawei-cloud.py huawei_get_cce_node_metrics
region=cn-north-4 cluster_id=<cluster-id>
node_ip=10.0.0.1 hours=1

Node GPU and xGPU metrics

Node的GPU和xGPU指标

python3 scripts/huawei-cloud.py huawei_get_cce_node_gpu_metrics
region=cn-north-4 cluster_id=<cluster-id>
node_ip=10.0.0.1 hours=1

undefined

python3 scripts/huawei-cloud.py huawei_get_cce_node_gpu_metrics
region=cn-north-4 cluster_id=<cluster-id>
node_ip=10.0.0.1 hours=1

undefined

3. CCE CoreDNS Metrics

3. CCE CoreDNS指标

bash

undefined

bash

undefined

CoreDNS key metrics: QPS, error rate excluding NXDOMAIN, NXDOMAIN rate, P95 latency, replicas, CPU, and memory

CoreDNS关键指标：QPS、排除NXDOMAIN的错误率、NXDOMAIN率、P95延迟、副本数、CPU和内存

python3 scripts/huawei-cloud.py huawei_get_cce_coredns_metrics
region=cn-north-4 cluster_id=<cluster-id>
namespace=kube-system pod_regex=".coredns." hours=1

undefined

python3 scripts/huawei-cloud.py huawei_get_cce_coredns_metrics
region=cn-north-4 cluster_id=<cluster-id>
namespace=kube-system pod_regex=".coredns." hours=1

undefined

4. CCE nginx-ingress Metrics

4. CCE nginx-ingress指标

bash

undefined

bash

undefined

nginx-ingress request processing and Ingress TLS certificate expiration

nginx-ingress请求处理及Ingress TLS证书过期情况

python3 scripts/huawei-cloud.py huawei_get_cce_nginx_ingress_metrics
region=cn-north-4 cluster_id=<cluster-id>
namespace=kube-system pod_regex=".*nginx.ingress.|.*ingress.nginx."
ingress_namespace=default cert_expire_warning_days=30 hours=1

undefined

undefined

5. CCE Autoscaler Metrics

5. CCE Autoscaler指标

bash

undefined

bash

undefined

Cluster Autoscaler and HPA metrics

集群Autoscaler和HPA指标

python3 scripts/huawei-cloud.py huawei_get_cce_autoscaler_metrics
region=cn-north-4 cluster_id=<cluster-id>
namespace=kube-system pod_regex=".*cluster.autoscaler.|.autoscaler."
include_hpa=true hours=1

undefined

undefined

6. Kubernetes Control Plane Metrics

6. Kubernetes控制面指标

bash

python3 scripts/huawei-cloud.py huawei_get_cce_apiserver_metrics \
  region=cn-north-4 cluster_id=<cluster-id> hours=1

python3 scripts/huawei-cloud.py huawei_get_cce_etcd_metrics \
  region=cn-north-4 cluster_id=<cluster-id> hours=1

python3 scripts/huawei-cloud.py huawei_get_cce_controller_manager_metrics \
  region=cn-north-4 cluster_id=<cluster-id> namespace=kube-system hours=1

python3 scripts/huawei-cloud.py huawei_get_cce_scheduler_metrics \
  region=cn-north-4 cluster_id=<cluster-id> namespace=kube-system hours=1

bash

python3 scripts/huawei-cloud.py huawei_get_cce_apiserver_metrics \
  region=cn-north-4 cluster_id=<cluster-id> hours=1

python3 scripts/huawei-cloud.py huawei_get_cce_etcd_metrics \
  region=cn-north-4 cluster_id=<cluster-id> hours=1

python3 scripts/huawei-cloud.py huawei_get_cce_controller_manager_metrics \
  region=cn-north-4 cluster_id=<cluster-id> namespace=kube-system hours=1

python3 scripts/huawei-cloud.py huawei_get_cce_scheduler_metrics \
  region=cn-north-4 cluster_id=<cluster-id> namespace=kube-system hours=1

7. Cloud Resource Metrics

7. 云资源指标

bash

undefined

bash

undefined

ECS instance metrics

ECS实例指标

python3 scripts/huawei-cloud.py huawei_get_ecs_metrics
region=cn-north-4 instance_id=<instance-id>

ELB metrics

ELB指标

python3 scripts/huawei-cloud.py huawei_get_elb_metrics
region=cn-north-4 elb_id=<loadbalancer-id> hours=1

EIP metrics

EIP指标

python3 scripts/huawei-cloud.py huawei_get_eip_metrics
region=cn-north-4 eip_id=<eip-id> hours=1

NAT Gateway metrics

NAT网关指标

python3 scripts/huawei-cloud.py huawei_get_nat_gateway_metrics
region=cn-north-4 nat_gateway_id=<nat-gateway-id> hours=1

undefined

python3 scripts/huawei-cloud.py huawei_get_nat_gateway_metrics
region=cn-north-4 nat_gateway_id=<nat-gateway-id> hours=1

undefined

8. Cluster Monitoring Aggregation

8. 集群监控聚合

bash

undefined

bash

undefined

Aggregate all monitoring data with anomaly detection

聚合所有监控数据并进行异常检测

python3 scripts/huawei-cloud.py huawei_cce_cluster_monitoring_aggregation
region=cn-north-4 cluster_id=<cluster-id>
start_time="2026-05-30 00:00:00" end_time="2026-05-30 23:59:59"
namespace=default top_n=10


This tool aggregates: Pod TopN CPU/memory, Node TopN CPU/memory/disk, ELB metrics (matched through LoadBalancer Services fetched by `kubectl`), NAT Gateway metrics, EIP metrics (bandwidth, packet loss), and anomaly detection using 80% threshold.

It also includes CoreDNS, nginx-ingress, and autoscaler summaries. Cloud resources are scoped to the current cluster when an association can be proven: ELB is matched through LoadBalancer Service IP/EIP, NAT Gateway is filtered by the cluster VPC, and EIP is limited to associated ELB/NAT/Service IPs.

LoadBalancer Service discovery uses `kubectl` with generated kubeconfig through the cluster EIP when external access is available. If the cluster has no EIP, it uses the `kubectl cce` plugin. If neither path works, aggregation fails.


该工具聚合以下内容：Pod TopN CPU/内存、Node TopN CPU/内存/磁盘、ELB指标（通过`kubectl`获取的LoadBalancer Services匹配）、NAT网关指标、EIP指标（带宽、丢包率），以及基于80%阈值的异常检测。

它还包含CoreDNS、nginx-ingress和autoscaler的摘要信息。当可以证明关联关系时，云资源会限定在当前集群范围内：ELB通过LoadBalancer Service IP/EIP匹配，NAT网关通过集群VPC过滤，EIP仅限关联的ELB/NAT/Service IP。

LoadBalancer Service发现会在外部可用时，通过集群EIP使用`kubectl`和生成的kubeconfig。如果集群没有EIP，则使用`kubectl cce`插件。如果两种方式都不可用，聚合操作会失败。

Risk Levels

风险等级

This skill is read-only. It does not create, update, delete, restart, scale, or modify Huawei Cloud or Kubernetes resources.

Level	Meaning	Execution Guidance
R3	No-risk read-only query or local analysis	May run automatically
R2	Low-risk change, such as creating monitoring configuration without deleting resources or increasing service capacity/cost	Not used by current tools
R1	Risky operation, such as restart-like impact, disabling protection, or changes that may increase cost or reduce observability	Not used by current tools
R0	Critical operation, such as deleting clusters, applications, or broad-impact monitoring protections	Not used by current tools

Tool	Operation Type	Risk Level	Description
`huawei_get_cce_pod_metrics_topN`	Query	R3	Read Pod CPU/memory/disk TopN metrics from AOM Prometheus
`huawei_get_cce_pod_metrics`	Query	R3	Read single Pod CPU/memory/disk time-series metrics
`huawei_get_cce_node_metrics_topN`	Query	R3	Read Node CPU/memory/disk TopN metrics from AOM Prometheus
`huawei_get_cce_node_metrics`	Query	R3	Read single Node CPU/memory/disk time-series metrics
`huawei_get_cce_node_gpu_metrics`	Query	R3	Read single Node GPU and xGPU metrics from AOM Prometheus
`huawei_get_cce_pod_gpu_metrics`	Query	R3	Read single Pod GPU and xGPU metrics from AOM Prometheus
`huawei_get_cce_coredns_metrics`	Query	R3	Read CoreDNS QPS, error rate excluding NXDOMAIN, NXDOMAIN rate, P95 latency, replicas, and per-Pod CPU/memory metrics
`huawei_get_cce_nginx_ingress_metrics`	Query	R3	Read nginx-ingress request-processing metrics and Ingress TLS certificate expiration status; QPS falls back to nginx process request counters when request-dimension metrics are absent
`huawei_get_cce_autoscaler_metrics`	Query	R3	Read Cluster Autoscaler scaling metrics, HPA replica state, and autoscaler Pod CPU/memory metrics
`huawei_get_cce_apiserver_metrics`	Query	R3	Read kube-apiserver QPS, error rate, latency, and inflight request metrics
`huawei_get_cce_etcd_metrics`	Query	R3	Read etcd leader, proposal, DB size, disk latency, CPU, and memory metrics
`huawei_get_cce_controller_manager_metrics`	Query	R3	Read control-plane workqueue depth, adds, retries, queue latency, and work duration metrics
`huawei_get_cce_scheduler_metrics`	Query	R3	Read scheduler attempts, pending Pods, scheduling latency, and queue metrics
`huawei_get_ecs_metrics`	Query	R3	Read ECS monitoring data through hcloud/CES
`huawei_get_elb_metrics`	Query	R3	Read ELB monitoring data through hcloud/CES
`huawei_get_eip_metrics`	Query	R3	Read EIP monitoring data through hcloud/CES
`huawei_get_nat_gateway_metrics`	Query	R3	Read NAT Gateway monitoring data through hcloud/CES
`huawei_cce_cluster_monitoring_aggregation`	Query + local analysis	R3	Aggregate Pod/Node/cloud-resource metrics and classify anomalies locally

本技能为只读模式。它不会创建、更新、删除、重启、扩缩容或修改华为云或Kubernetes资源。

等级	含义	执行指导
R3	无风险的只读查询或本地分析	可自动运行
R2	低风险变更，例如创建监控配置而不删除资源或增加服务容量/成本	当前工具未使用
R1	有风险的操作，例如类似重启的影响、禁用保护或可能增加成本或降低可观测性的变更	当前工具未使用
R0	关键操作，例如删除集群、应用或影响范围广的监控保护	当前工具未使用

工具	操作类型	风险等级	描述
`huawei_get_cce_pod_metrics_topN`	查询	R3	从AOM Prometheus读取Pod CPU/内存/磁盘TopN指标
`huawei_get_cce_pod_metrics`	查询	R3	读取单个Pod的CPU/内存/磁盘时间序列指标
`huawei_get_cce_node_metrics_topN`	查询	R3	从AOM Prometheus读取Node CPU/内存/磁盘TopN指标
`huawei_get_cce_node_metrics`	查询	R3	读取单个Node的CPU/内存/磁盘时间序列指标
`huawei_get_cce_node_gpu_metrics`	查询	R3	从AOM Prometheus读取单个Node的GPU和xGPU指标
`huawei_get_cce_pod_gpu_metrics`	查询	R3	从AOM Prometheus读取单个Pod的GPU和xGPU指标
`huawei_get_cce_coredns_metrics`	查询	R3	读取CoreDNS的QPS、排除NXDOMAIN的错误率、NXDOMAIN率、P95延迟、副本数及每个Pod的CPU/内存指标
`huawei_get_cce_nginx_ingress_metrics`	查询	R3	读取nginx-ingress的请求处理指标和Ingress TLS证书过期状态；当请求维度指标缺失时，QPS会回退到nginx进程请求计数器（如果可用）
`huawei_get_cce_autoscaler_metrics`	查询	R3	读取集群Autoscaler的扩缩容指标、HPA副本状态及autoscaler Pod的CPU/内存指标
`huawei_get_cce_apiserver_metrics`	查询	R3	读取kube-apiserver的QPS、错误率、延迟及进行中请求指标
`huawei_get_cce_etcd_metrics`	查询	R3	读取etcd的领导者状态、提案、数据库大小、磁盘延迟、CPU和内存指标
`huawei_get_cce_controller_manager_metrics`	查询	R3	读取控制面工作队列深度、添加数、重试数、队列延迟及工作时长指标
`huawei_get_cce_scheduler_metrics`	查询	R3	读取调度器尝试次数、待调度Pod数、调度延迟及队列指标
`huawei_get_ecs_metrics`	查询	R3	通过hcloud/CES读取ECS监控数据
`huawei_get_elb_metrics`	查询	R3	通过hcloud/CES读取ELB监控数据
`huawei_get_eip_metrics`	查询	R3	通过hcloud/CES读取EIP监控数据
`huawei_get_nat_gateway_metrics`	查询	R3	通过hcloud/CES读取NAT网关监控数据
`huawei_cce_cluster_monitoring_aggregation`	查询 + 本地分析	R3	聚合Pod/Node/云资源指标并在本地进行异常分类

Parameter Reference

参数参考

Common Parameters

通用参数

Parameter	Required/Optional	Description	Default
`region`	Required	Huawei Cloud region	`HUAWEI_REGION`
`cluster_id`	Required	CCE cluster ID	N/A
`namespace`	Recommended	Kubernetes namespace	`default`
`ak`	Optional	Explicit AK; highest priority for all calls	profile/env fallback
`sk`	Optional	Explicit SK; highest priority for all calls	profile/env fallback
`project_id`	Optional	Explicit Project ID; hcloud uses profile before env fallback	Auto from IAM/profile

参数	必填/可选	描述	默认值
`region`	必填	华为云区域	`HUAWEI_REGION`
`cluster_id`	必填	CCE集群ID	N/A
`namespace`	推荐	Kubernetes命名空间	`default`
`ak`	可选	显式AK；所有调用的最高优先级	配置文件/环境变量备选
`sk`	可选	显式SK；所有调用的最高优先级	配置文件/环境变量备选
`project_id`	可选	显式项目ID；hcloud优先使用配置文件，其次是环境变量备选	从IAM/配置文件自动获取

huawei_get_cce_pod_metrics_topN

Parameters

huawei_get_cce_pod_metrics_topN

参数

Parameter	Required	Description	Default
`namespace`	No	Namespace filter	all
`label_selector`	No	Label selector (e.g. app=web)	N/A
`top_n`	No	Number of top items	10
`hours`	No	Metrics lookback hours	1
`node_ip`	No	Filter Pods on specific node	N/A
`cpu_query`	No	Custom CPU PromQL	Auto
`memory_query`	No	Custom memory PromQL	Auto
`disk_query`	No	Custom disk PromQL	Auto

参数	必填	描述	默认值
`namespace`	否	命名空间筛选	全部
`label_selector`	否	标签选择器（例如app=web）	N/A
`top_n`	否	TopN条目数量	10
`hours`	否	指标回溯小时数	1
`node_ip`	否	筛选特定节点上的Pod	N/A
`cpu_query`	否	自定义CPU PromQL	自动生成
`memory_query`	否	自定义内存PromQL	自动生成
`disk_query`	否	自定义磁盘PromQL	自动生成

huawei_get_cce_pod_metrics

Parameters

huawei_get_cce_pod_metrics

参数

Parameter	Required	Description	Default
`pod_name`	Yes	Target Pod name	N/A
`namespace`	No	Namespace	`default`
`hours`	No	Metrics lookback hours	1
`cpu_query`	No	Custom CPU PromQL	Auto
`memory_query`	No	Custom memory PromQL	Auto
`disk_query`	No	Custom disk PromQL	Auto

参数	必填	描述	默认值
`pod_name`	是	目标Pod名称	N/A
`namespace`	否	命名空间	`default`
`hours`	否	指标回溯小时数	1
`cpu_query`	否	自定义CPU PromQL	自动生成
`memory_query`	否	自定义内存PromQL	自动生成
`disk_query`	否	自定义磁盘PromQL	自动生成

huawei_get_cce_pod_gpu_metrics

Parameters

huawei_get_cce_pod_gpu_metrics

参数

Parameter	Required	Description	Default
`pod_name`	Yes	Target Pod name	N/A
`namespace`	No	Target Pod namespace	all
`hours`	No	Metrics lookback hours	1
`gpu_selector`	No	Custom GPU metric label selector. Use this when GPU metrics do not use the `pod` or `namespace` labels	`pod="<pod_name>",namespace="<namespace>"`

Optional custom PromQL overrides are supported for GPU utilization, memory, schedule policy, xGPU allocation/usage, and xGPU health metrics.

参数	必填	描述	默认值
`pod_name`	是	目标Pod名称	N/A
`namespace`	否	目标Pod命名空间	全部
`hours`	否	指标回溯小时数	1
`gpu_selector`	否	自定义GPU指标标签选择器。当GPU指标不使用 `pod` 或 `namespace` 标签时使用	`pod="<pod_name>",namespace="<namespace>"`

支持对GPU利用率、内存、调度策略、xGPU分配/使用情况及xGPU健康指标进行可选的自定义PromQL覆盖。

huawei_get_cce_node_metrics_topN

Parameters

huawei_get_cce_node_metrics_topN

参数

Parameter	Required	Description	Default
`top_n`	No	Number of top items	10
`hours`	No	Metrics lookback hours	1

参数	必填	描述	默认值
`top_n`	否	TopN条目数量	10
`hours`	否	指标回溯小时数	1

huawei_get_cce_node_metrics

Parameters

huawei_get_cce_node_metrics

参数

Parameter	Required	Description	Default
`node_ip`	Yes	Target Node IP	N/A
`hours`	No	Metrics lookback hours	1

参数	必填	描述	默认值
`node_ip`	是	目标Node IP	N/A
`hours`	否	指标回溯小时数	1

huawei_get_cce_node_gpu_metrics

Parameters

huawei_get_cce_node_gpu_metrics

参数

Parameter	Required	Description	Default
`node_ip`	Yes	Target Node IP or node name	N/A
`hours`	No	Metrics lookback hours	1
`gpu_selector`	No	Custom GPU metric label selector. Use this when GPU metrics do not use the `node` label	`node=~"<node_ip>

Optional custom PromQL overrides are supported for GPU utilization, memory, temperature, power, schedule policy, xGPU allocation/usage, and xGPU health metrics.

参数	必填	描述	默认值
`node_ip`	是	目标Node IP或节点名称	N/A
`hours`	否	指标回溯小时数	1
`gpu_selector`	否	自定义GPU指标标签选择器。当GPU指标不使用 `node` 标签时使用	`node=~"<node_ip>

支持对GPU利用率、内存、温度、功耗、调度策略、xGPU分配/使用情况及xGPU健康指标进行可选的自定义PromQL覆盖。

huawei_get_cce_coredns_metrics

Parameters

huawei_get_cce_coredns_metrics

参数

Parameter	Required	Description	Default
`namespace`	No	CoreDNS namespace	`kube-system`
`pod_regex`	No	Regex used to match CoreDNS Pods	`.coredns.`
`hours`	No	Metrics lookback hours	1

Optional custom PromQL overrides are supported for QPS, error rate, NXDOMAIN rate, P95 latency, CPU, memory, and replica count.

参数	必填	描述	默认值
`namespace`	否	CoreDNS命名空间	`kube-system`
`pod_regex`	否	用于匹配CoreDNS Pod的正则表达式	`.coredns.`
`hours`	否	指标回溯小时数	1

支持对QPS、错误率、NXDOMAIN率、P95延迟、CPU、内存及副本数进行可选的自定义PromQL覆盖。

huawei_get_cce_nginx_ingress_metrics

Parameters

huawei_get_cce_nginx_ingress_metrics

参数

Parameter	Required	Description	Default
`namespace`	No	Namespace of nginx-ingress controller Pods. Use an empty value to query all namespaces	`kube-system`
`pod_regex`	No	Regex used to match nginx-ingress controller Pods	`.*nginx.ingress.
`ingress_namespace`	No	Namespace filter for Ingress TLS certificate checks	all
`hours`	No	Metrics lookback hours	1
`cert_expire_warning_days`	No	Days before expiry to mark certificates as warning	30
`check_certificates`	No	Whether to inspect Ingress TLS Secrets for expiration status	true

Ingress-controller metrics depend on the corresponding AOM PodMonitor. The

nginx_ingress_controller_requests

metric must be explicitly allowed in the ingress-controller PodMonitor; otherwise request-dimension metrics such as 4xx/5xx QPS, success rate, and latency may be empty, and QPS may only use the

nginx_ingress_controller_nginx_process_requests_total

fallback when available.

Optional custom PromQL overrides are supported for QPS, 4xx/5xx, success rate, P95 latency, active connections, CPU, and memory.

参数	必填	描述	默认值
`namespace`	否	nginx-ingress控制器Pod的命名空间。使用空值查询所有命名空间	`kube-system`
`pod_regex`	否	用于匹配nginx-ingress控制器Pod的正则表达式	`.*nginx.ingress.
`ingress_namespace`	否	Ingress TLS证书检查的命名空间筛选	全部
`hours`	否	指标回溯小时数	1
`cert_expire_warning_days`	否	证书过期前标记为警告的天数	30
`check_certificates`	否	是否检查Ingress TLS Secrets的过期状态	true

Ingress控制器指标依赖对应的AOM PodMonitor。必须在ingress控制器的PodMonitor中明确允许

nginx_ingress_controller_requests

指标；否则4xx/5xx QPS、成功率、延迟等请求维度指标可能为空，且QPS可能仅在可用时使用

nginx_ingress_controller_nginx_process_requests_total

作为回退。

支持对QPS、4xx/5xx错误率、成功率、P95延迟、活跃连接数、CPU及内存进行可选的自定义PromQL覆盖。

huawei_get_cce_autoscaler_metrics

Parameters

huawei_get_cce_autoscaler_metrics

参数

Parameter	Required	Description	Default
`namespace`	No	Namespace of Cluster Autoscaler Pods. Use an empty value to query all namespaces	`kube-system`
`pod_regex`	No	Regex used to match autoscaler Pods	`.*cluster.autoscaler.
`hpa_namespace`	No	Namespace filter for HPA replica metrics	all
`hours`	No	Metrics lookback hours	1
`include_hpa`	No	Whether to query HPA current/desired replica metrics	true

Optional custom PromQL overrides are supported for unschedulable Pods, node states, scale events, errors, node groups, HPA replicas, CPU, and memory.

参数	必填	描述	默认值
`namespace`	否	集群Autoscaler Pod的命名空间。使用空值查询所有命名空间	`kube-system`
`pod_regex`	否	用于匹配autoscaler Pod的正则表达式	`.*cluster.autoscaler.
`hpa_namespace`	否	HPA副本指标的命名空间筛选	全部
`hours`	否	指标回溯小时数	1
`include_hpa`	否	是否查询HPA当前/期望副本指标	true

支持对不可调度Pod数、节点状态、扩缩容事件、错误信息、节点组、HPA副本数、CPU及内存进行可选的自定义PromQL覆盖。

Kubernetes Control Plane Tool Parameters

Kubernetes控制面工具参数

Applies to

huawei_get_cce_apiserver_metrics

huawei_get_cce_etcd_metrics

huawei_get_cce_controller_manager_metrics

, and

huawei_get_cce_scheduler_metrics

huawei_get_cce_apiserver_metrics

defaults to

cluster="<cluster_id>",component="apiserver"

and does not add namespace or Pod labels. Its default P95 latency excludes

WATCH|CONNECT

requests and also returns

latency_p95_by_verb_ms

for diagnosis. Use

metric_selector

only when the Prometheus labels differ.

huawei_get_cce_etcd_metrics

defaults to

cluster="<cluster_id>"

and does not add namespace or Pod labels. Use

metric_selector

only when the Prometheus labels differ.

huawei_get_cce_controller_manager_metrics

defaults to

cluster="<cluster_id>"

because CCE AOM workqueue metrics may not expose stable controller-manager Pod labels. It returns both aggregate workqueue metrics and per-queue

name

breakdowns.

huawei_get_cce_scheduler_metrics

defaults to

cluster="<cluster_id>"

and returns aggregate metrics plus

result

profile/result

, and

queue

breakdowns.

Controller-manager, scheduler, and etcd metrics depend on AOM ServiceMonitor collection being enabled for the corresponding

kube-controller-manager

kube-scheduler

, and

etcd-server

endpoints. If ServiceMonitor is not enabled, the tools can run successfully but return empty series.

Parameter	Required	Description	Default
`namespace`	No	Namespace of control-plane Pods. Use an empty value to query all namespaces	`kube-system`
`pod_regex`	No	Regex used to match target component Pods	component-specific
`metric_selector`	No	Custom apiserver/etcd/controller-manager/scheduler metric label selector	apiserver: `cluster="<cluster_id>",component="apiserver"` ; etcd/controller-manager/scheduler: `cluster="<cluster_id>"`
`hours`	No	Metrics lookback hours	1

适用于

huawei_get_cce_apiserver_metrics

、

huawei_get_cce_etcd_metrics

、

huawei_get_cce_controller_manager_metrics

和

huawei_get_cce_scheduler_metrics

。

huawei_get_cce_apiserver_metrics

默认使用

cluster="<cluster_id>",component="apiserver"

，不会添加命名空间或Pod标签。其默认P95延迟排除

WATCH|CONNECT

请求，同时返回

latency_p95_by_verb_ms

用于诊断。仅当Prometheus标签不同时使用

metric_selector

。

huawei_get_cce_etcd_metrics

默认使用

cluster="<cluster_id>"

，不会添加命名空间或Pod标签。仅当Prometheus标签不同时使用

metric_selector

。

huawei_get_cce_controller_manager_metrics

默认使用

cluster="<cluster_id>"

，因为CCE AOM工作队列指标可能不会暴露稳定的controller-manager Pod标签。它返回聚合工作队列指标和按

name

拆分的每个队列详情。

huawei_get_cce_scheduler_metrics

默认使用

cluster="<cluster_id>"

，返回聚合指标以及按

result

、

profile/result

和

queue

拆分的详情。

Controller-manager、scheduler和etcd指标依赖为对应的

kube-controller-manager

、

kube-scheduler

和

etcd-server

端点启用AOM ServiceMonitor收集。如果未启用ServiceMonitor，工具可以成功运行但返回空序列。

参数	必填	描述	默认值
`namespace`	否	控制面Pod的命名空间。使用空值查询所有命名空间	`kube-system`
`pod_regex`	否	用于匹配目标组件Pod的正则表达式	组件特定值
`metric_selector`	否	自定义apiserver/etcd/controller-manager/scheduler指标标签选择器	apiserver: `cluster="<cluster_id>",component="apiserver"` ; etcd/controller-manager/scheduler: `cluster="<cluster_id>"`
`hours`	否	指标回溯小时数	1

Cloud Resource Tool Parameters

云资源工具参数

Tool	Required ID Parameter	Optional Parameters
`huawei_get_ecs_metrics`	`instance_id`	none
`huawei_get_elb_metrics`	`elb_id`	`hours`
`huawei_get_eip_metrics`	`eip_id`	`hours`
`huawei_get_nat_gateway_metrics`	`nat_gateway_id`	`hours`

工具	必填ID参数	可选参数
`huawei_get_ecs_metrics`	`instance_id`	无
`huawei_get_elb_metrics`	`elb_id`	`hours`
`huawei_get_eip_metrics`	`eip_id`	`hours`
`huawei_get_nat_gateway_metrics`	`nat_gateway_id`	`hours`

huawei_cce_cluster_monitoring_aggregation

Parameters

huawei_cce_cluster_monitoring_aggregation

参数

Parameter	Required	Description	Default
`start_time`	Yes	Start time (YYYY-MM-DD HH:MM:SS)	N/A
`end_time`	Yes	End time (YYYY-MM-DD HH:MM:SS)	N/A
`namespace`	No	Namespace filter	`default`
`top_n`	No	Number of top items	10
`security_token`	No	Temporary security token for AK/SK session credentials	env fallback

参数	必填	描述	默认值
`start_time`	是	开始时间（YYYY-MM-DD HH:MM:SS)	N/A
`end_time`	是	结束时间（YYYY-MM-DD HH:MM:SS）	N/A
`namespace`	否	命名空间筛选	`default`
`top_n`	否	TopN条目数量	10
`security_token`	否	AK/SK会话凭证的临时安全令牌	环境变量备选

Output Format

输出格式

See Output Schema for the complete JSON response structure.

Key output fields:

```
success
```
— boolean, true if query completed
```
region
```
— Huawei Cloud region
```
cluster_id
```
/
```
cluster_name
```
— CCE cluster identity
```
aom_instance_id
```
— AOM Prometheus instance used for metric queries
```
metrics
```
— Dict with cpu/memory/disk data per resource, including status classification
```
certificate_check
```
— nginx-ingress Ingress TLS certificate expiration summary when certificate checking is enabled
```
time_series
```
— Historical data points with
```
timestamp
```
,
```
time
```
,
```
average
```
,
```
min
```
,
```
max
```
```
status
```
— Threshold classification:
```
critical
```
(>80% CPU, >85% memory/disk),
```
warning
```
(>50% CPU/memory, >70% disk),
```
normal
```
(below warning),
```
unknown
```
(no data)

完整的JSON响应结构请参见输出Schema。

关键输出字段:

```
success
```
— 布尔值，查询完成时为true
```
region
```
— 华为云区域
```
cluster_id
```
/
```
cluster_name
```
— CCE集群标识
```
aom_instance_id
```
— 用于指标查询的AOM Prometheus实例
```
metrics
```
— 每个资源的cpu/内存/磁盘数据字典，包含状态分类
```
certificate_check
```
— 启用证书检查时，nginx-ingress的Ingress TLS证书过期摘要
```
time_series
```
— 包含
```
timestamp
```
、
```
time
```
、
```
average
```
、
```
min
```
、
```
max
```
的历史数据点
```
status
```
— 阈值分类：
```
critical
```
（CPU>80%，内存/磁盘>85%）、
```
warning
```
（CPU/内存>50%，磁盘>70%）、
```
normal
```
（低于警告阈值）、
```
unknown
```
（无数据）

Workflow

工作流程

Resolve region, cluster ID, and credentials using the documented priority.
Discover the AOM Prometheus instance from the CCE cluster add-on binding.
Start with Pod/Node TopN or aggregation, then drill into a Pod, Node, component, or cloud resource.
Keep PromQL scoped by
```
cluster="<cluster_id>"
```
; add namespace, pod, or resource filters only to reduce noise.
Use status classification as an investigation lead, then correlate anomalies with events or alarm history.

使用文档中说明的优先级解析区域、集群ID和凭证。
从CCE集群插件绑定中发现AOM Prometheus实例。
从Pod/Node TopN或聚合开始，然后深入到单个Pod、Node、组件或云资源。
使用
```
cluster="<cluster_id>"
```
限定PromQL范围；仅在需要减少干扰时添加命名空间、Pod或资源筛选器。
将状态分类作为调查线索，然后将异常与事件或告警历史关联起来。

Verification

验证

Run

python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics_topN region=cn-north-4 cluster_id=<cluster-id> namespace=default top_n=5

to verify Pod metric queries

Run

python3 scripts/huawei-cloud.py huawei_get_cce_node_metrics_topN region=cn-north-4 cluster_id=<cluster-id> top_n=5

to verify Node metric queries

Run

python3 scripts/huawei-cloud.py huawei_get_ecs_metrics region=cn-north-4 instance_id=<instance-id>

to verify CES metric connectivity

运行

python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics_topN region=cn-north-4 cluster_id=<cluster-id> namespace=default top_n=5

验证Pod指标查询

运行

python3 scripts/huawei-cloud.py huawei_get_cce_node_metrics_topN region=cn-north-4 cluster_id=<cluster-id> top_n=5

验证Node指标查询

运行

python3 scripts/huawei-cloud.py huawei_get_ecs_metrics region=cn-north-4 instance_id=<instance-id>

验证CES指标连通性

Best Practices

最佳实践

Start with Pod/Node TopN before drilling into individual resources.
Keep
```
hours
```
small (1-4) for recent analysis; cap historical reviews at 24 hours.
Provide
```
namespace
```
to reduce Pod noise while preserving the cluster filter.
Focus on
```
critical
```
and
```
warning
```
resources first.

Use

huawei_cce_cluster_monitoring_aggregation

for full-cluster health checks.

Correlate metric anomalies with

huawei-cloud-cce-kubernetes-event-analyzer

Do not expose production Pod names, node IPs, or cluster IDs in public summaries.

在深入单个资源之前，先查看Pod/Node TopN。
近期分析时将
```
hours
```
设置为较小值（1-4）；历史回顾最多限制为24小时。
提供
```
namespace
```
以减少Pod干扰，同时保留集群筛选。
优先关注
```
critical
```
和
```
warning
```
级别的资源。

使用

huawei_cce_cluster_monitoring_aggregation

进行全集群健康检查。

将指标异常与

huawei-cloud-cce-kubernetes-event-analyzer

关联分析。

请勿在公开摘要中暴露生产环境的Pod名称、节点IP或集群ID。

Notes

注意事项

This skill is strictly read-only and never modifies resources or configurations.
Thresholds are predefined baselines; tune them against workload SLOs before making operational decisions.
AK/SK must never be hardcoded; use hcloud profile for normal hcloud calls or environment fallback for signed AOM/Kubernetes calls.
```
scripts/huawei-cloud.py
```
is the only user-facing execution method
AOM Prometheus instance is auto-discovered; no need to manually specify
```
aom_instance_id
```
Cloud resource metrics (ECS/ELB/EIP/NAT) use CES (Cloud Eye Service), not AOM
Do not make automatic scaling or remediation decisions based solely on metric analysis.

本技能严格为只读模式，绝不会修改资源或配置。
阈值为预定义基线；在做出操作决策之前，请根据工作负载SLO调整阈值。
AK/SK绝不能硬编码；正常hcloud调用使用hcloud配置文件，签名AOM/Kubernetes调用使用环境变量备选。
```
scripts/huawei-cloud.py
```
是唯一面向用户的执行方式
AOM Prometheus实例会自动发现；无需手动指定
```
aom_instance_id
```
云资源指标（ECS/ELB/EIP/NAT）使用CES（云眼服务），而非AOM
请勿仅基于指标分析做出自动扩缩容或修复决策。

Troubleshooting

故障排除

Pitfall	Symptom	Quick Fix
Missing `cluster_id`	Action fails immediately	Provide `cluster_id` from cluster listing
AOM Prometheus instance not found	Metric queries return empty results	Ensure AOM Prom instance is created for the cluster; check `aom:instance:list` permission
Large time window without namespace filter	Slow response, too many results	Narrow `hours` to 1-4 and add `namespace` filter
Cloud resource ID not found	ECS/ELB/EIP/NAT query returns error	Verify resource ID and CES IAM permission
Custom PromQL syntax error	Custom query returns empty	Use default PromQL unless familiar with AOM PromQL
Aggregation missing time range	`start_time` / `end_time` missing	Provide both time boundaries

问题	症状	快速修复
缺少 `cluster_id`	操作立即失败	从集群列表中提供 `cluster_id`
未找到AOM Prometheus实例	指标查询返回空结果	确保为集群创建了AOM Prom实例；检查 `aom:instance:list` 权限
未设置命名空间筛选的大时间窗口	响应缓慢、结果过多	将 `hours` 缩小到1-4并添加 `namespace` 筛选
云资源ID未找到	ECS/ELB/EIP/NAT查询返回错误	验证资源ID和CES IAM权限
自定义PromQL语法错误	自定义查询返回空结果	除非熟悉AOM PromQL，否则使用默认PromQL
聚合缺少时间范围	`start_time` / `end_time` 缺失	提供两个时间边界

Limitations

局限性

AOM Prometheus data requires the cluster Prometheus add-on to be integrated with AOM.
Control-plane ServiceMonitors and component PodMonitors must be enabled before related metrics appear.
Query results reflect collected monitoring data only; missing series are not proof that the workload is healthy.
This skill does not remediate, scale, restart, create, update, or delete cloud or Kubernetes resources.

AOM Prometheus数据要求集群Prometheus插件与AOM集成。
控制面ServiceMonitors和组件PodMonitors必须启用，相关指标才会显示。
查询结果仅反映收集到的监控数据；缺少序列并不代表工作负载健康。
本技能不会修复、扩缩容、重启、创建、更新或删除云或Kubernetes资源。

References

参考文档

Document	Description
Workflow	Metric query sequence, threshold detection, next-step handoff
Risk Rules	Read-only constraints, data redaction, time-bounding, threshold caveats
Output Schema	JSON response schema for metric and status output
CLI Installation Guide	hcloud, kubectl, kubectl-cce, and dispatcher setup
IAM Policies	Required read-only Huawei Cloud and Kubernetes permissions
Verification Method	Static checks and smoke tests
Acceptance Criteria	Functional, security, documentation, and quality gates

文档	描述
工作流程	指标查询序列、阈值检测、下一步交接
风险规则	只读约束、数据编辑、时间限制、阈值注意事项
输出Schema	指标和状态输出的JSON响应Schema
CLI安装指南	hcloud、kubectl、kubectl-cce和调度器的设置
IAM策略	所需的只读华为云和Kubernetes权限
验证方法	静态检查和冒烟测试
验收标准	功能、安全、文档和质量门控

huawei-cloud-cce-metric-analyzer

Original

Translation

Huawei Cloud CCE Metric Analyzer

华为云CCE指标分析器

Overview

概述

Prerequisites

前提条件

1. Runtime Dependencies

1. 运行时依赖

2. Credential Configuration

2. 凭证配置

3. IAM Permission Requirements

3. IAM权限要求

Core Commands

核心命令

KooCLI命令格式标准

KooCLI命令格式标准

1. CCE Pod Metrics

1. CCE Pod指标

Pod TopN — cluster-wide CPU/memory ranking

Pod TopN — 集群范围的CPU/内存排名

Pod TopN with label selector

带标签选择器的Pod TopN

Single Pod time-series

单个Pod的时间序列

Single Pod GPU and xGPU metrics

单个Pod的GPU和xGPU指标

2. CCE Node Metrics

2. CCE Node指标

Node TopN — cluster-wide CPU/memory/disk ranking

Node TopN — 集群范围的CPU/内存/磁盘排名

Single Node time-series

单个Node的时间序列

Node GPU and xGPU metrics

Node的GPU和xGPU指标

3. CCE CoreDNS Metrics

3. CCE CoreDNS指标

CoreDNS key metrics: QPS, error rate excluding NXDOMAIN, NXDOMAIN rate, P95 latency, replicas, CPU, and memory

CoreDNS关键指标：QPS、排除NXDOMAIN的错误率、NXDOMAIN率、P95延迟、副本数、CPU和内存

4. CCE nginx-ingress Metrics

4. CCE nginx-ingress指标

nginx-ingress request processing and Ingress TLS certificate expiration

nginx-ingress请求处理及Ingress TLS证书过期情况

5. CCE Autoscaler Metrics

5. CCE Autoscaler指标

Cluster Autoscaler and HPA metrics

集群Autoscaler和HPA指标

6. Kubernetes Control Plane Metrics

6. Kubernetes控制面指标

7. Cloud Resource Metrics

7. 云资源指标

ECS instance metrics

ECS实例指标

ELB metrics

ELB指标

EIP metrics

EIP指标

NAT Gateway metrics

NAT网关指标

8. Cluster Monitoring Aggregation

8. 集群监控聚合

Aggregate all monitoring data with anomaly detection

聚合所有监控数据并进行异常检测

Risk Levels

风险等级

Parameter Reference

参数参考

Common Parameters

通用参数

huawei_get_cce_pod_metrics_topN Parameters

huawei_get_cce_pod_metrics_topN参数

huawei_get_cce_pod_metrics Parameters

huawei_get_cce_pod_metrics参数

huawei_get_cce_pod_gpu_metrics Parameters

huawei_get_cce_pod_gpu_metrics参数

huawei_get_cce_node_metrics_topN Parameters

huawei_get_cce_node_metrics_topN参数

huawei_get_cce_node_metrics Parameters

`huawei_get_cce_pod_metrics_topN`
Parameters

`huawei_get_cce_pod_metrics_topN`
参数

`huawei_get_cce_pod_metrics`
Parameters

`huawei_get_cce_pod_metrics`
参数

`huawei_get_cce_pod_gpu_metrics`
Parameters

`huawei_get_cce_pod_gpu_metrics`
参数

`huawei_get_cce_node_metrics_topN`
Parameters

`huawei_get_cce_node_metrics_topN`
参数

`huawei_get_cce_node_metrics`
Parameters

`huawei_get_cce_node_metrics`
参数

`huawei_get_cce_node_gpu_metrics`
Parameters

`huawei_get_cce_node_gpu_metrics`
参数

`huawei_get_cce_coredns_metrics`
Parameters

`huawei_get_cce_coredns_metrics`
参数

`huawei_get_cce_nginx_ingress_metrics`
Parameters

`huawei_get_cce_nginx_ingress_metrics`
参数

`huawei_get_cce_autoscaler_metrics`
Parameters

`huawei_get_cce_autoscaler_metrics`
参数

`huawei_cce_cluster_monitoring_aggregation`
Parameters

`huawei_cce_cluster_monitoring_aggregation`
参数