iap-agent-platform-troubleshooting

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

IAP troubleshooting for Gemini Enterprise Agent Platform

Gemini Enterprise Agent Platform 的 IAP 故障排查

Find out why Identity-Aware Proxy (IAP) denies a request from a Gemini Enterprise Agent Platform agent, and print the smallest fix for the user to run. This skill supports Agent Runtime (Reasoning Engine) agents that use agent identity and send traffic through Agent Gateway.
查明 Identity-Aware Proxy (IAP) 拒绝 Gemini Enterprise Agent Platform Agent 请求的原因,并输出供用户运行的最小修复方案。
此 skill 支持使用 Agent identity 并通过 Agent Gateway 发送流量的 Agent Runtime(Reasoning Engine)Agent。

Operating contract

操作约定

  • Read only. Run only the read commands in this skill and its references (
    describe
    ,
    list
    ,
    get-iam-policy
    ,
    search-target-policy-bindings
    , and
    logging read
    ), or the matching read-only MCP tools (see Tool selection). The setup commands in prerequisites.md change only local CLI settings.
  • Print fixes, never run them. Don't run a command that creates, imports, updates, binds, enables, or deletes anything, even if the user asks you to. Print the remediation template for the root cause (see Report), and let the user run it. Access policies and IAP allow policies apply to every agent whose traffic goes through Agent Gateway in that project, so a wrong change can block or expose other agents.
  • Evidence first. Copy principals, resource names, and hostnames from command output. Don't build them from memory.
  • Stop on missing access. If a read fails with
    PERMISSION_DENIED
    , report the missing role instead of working around it.
  • 只读。 仅运行本 skill 及其参考文件中的只读命令(
    describe
    、
    list
    、
    get-iam-policy
    、
    search-target-policy-bindings
    和
    logging read
    ),或对应的只读 MCP 工具(见工具选择)。prerequisites.md 中的设置命令仅会更改本地 CLI 设置。
  • 只输出修复方案,绝不执行。 不要运行任何用于创建、导入、更新、绑定、启用或删除资源的命令,即使用户要求这样做。应输出与根因对应的修复模板(见报告),由用户自行运行。Access policies 和 IAP allow policies 会应用于该项目内所有通过 Agent Gateway 发送流量的 Agent,因此错误变更可能阻断或暴露其他 Agent。
  • 以证据为先。 从命令输出中复制 principal、resource name 和 hostname。不要凭记忆构建这些值。
  • 缺少访问权限时停止。 如果读取操作因
    PERMISSION_DENIED
    失败,请报告缺少的角色,不要尝试绕过权限检查。

Attribution

归因

MUST READ: Before executing any
gcloud
, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix
gcloud
with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:iap-agent-platform-troubleshooting)"
and append
agent-skills/1.0 (skill:iap-agent-platform-troubleshooting)
to
User-Agent
headers.
必须阅读:在执行任何
gcloud
、REST API 或 MCP 命令之前,必须阅读 Usage Attribution。对于直接在终端中执行的命令(绝不用于生成的脚本、Terraform 或用户代码),请在
gcloud
前添加
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:iap-agent-platform-troubleshooting)"
,并在
User-Agent
headers 中附加
agent-skills/1.0 (skill:iap-agent-platform-troubleshooting)
。

Tool selection

工具选择

Prefer the read tools of a connected Google Cloud remote MCP server over the matching
gcloud
read, because remote MCP servers apply IAM, authentication, and audit logging to every call. Use
gcloud
for any read that no connected MCP server covers.
  • Cloud Logging: run the log queries with
    list_log_entries
    . The queries in the references use
    gcloud
    syntax, so adapt each one: set
    resourceNames
    to
    projects/{project_id}
    , keep the filter, replace
    --freshness
    with a
    timestamp>=
    clause in the filter, and set
    orderBy
    to
    timestamp desc
    and
    pageSize
    to the
    --limit
    value. Don't pass
    gcloud
    flags such as
    --freshness
    to the tool.
  • Agent Registry: resolve destinations with its read tools, such as
    list_endpoints
    ,
    get_endpoint
    ,
    list_mcp_servers
    , and
    get_mcp_server
    .
  • Cloud CLI: run the
    gcloud
    reads in this skill through
    run_gcloud_command
    .
  • IAP, Agent Gateway, and authorization policies: no remote MCP server covers these, so read them with
    gcloud
    , directly or through the Cloud CLI MCP server.
The read-only contract covers MCP tools too: never call a tool that creates, updates, or deletes anything, such as
create_binding
or
delete_service
.
优先使用已连接的 Google Cloud 远程 MCP server 的只读工具,而不是对应的
gcloud
只读操作,因为远程 MCP server 会为每次调用应用 IAM、身份验证和审计日志记录。对于任何已连接 MCP server 未覆盖的只读操作,使用
gcloud
。
  • Cloud Logging: 使用
    list_log_entries
    运行日志查询。参考文件中的查询采用
    gcloud
    语法,因此需要逐条调整:将
    resourceNames
    设置为
    projects/{project_id}
    ,保留 filter,将
    --freshness
    替换为 filter 中的
    timestamp>=
    子句,并将
    orderBy
    设置为
    timestamp desc
    ,将
    pageSize
    设置为
    --limit
    的值。不要向该工具传递
    --freshness
    等
    gcloud
    flags。
  • Agent Registry: 使用其只读工具解析目标,例如
    list_endpoints
    、
    get_endpoint
    、
    list_mcp_servers
    和
    get_mcp_server
    。
  • Cloud CLI: 通过
    run_gcloud_command
    运行此 skill 中的
    gcloud
    只读操作。
  • IAP、Agent Gateway 和授权策略: 没有远程 MCP server 覆盖这些资源,因此直接使用
    gcloud
    ,或通过 Cloud CLI MCP server 读取。
只读约定同样适用于 MCP 工具:绝不调用任何创建、更新或删除资源的工具,例如
create_binding
或
delete_service
。

Prerequisites

前置条件

The Google Cloud CLI must be installed and authenticated for
{project_id}
. Load prerequisites.md if it isn't, or when a read fails with
PERMISSION_DENIED
or a disabled API. It covers
gcloud auth login
, the APIs, the read-only roles, billing, and the default region (
us-central1
). The diagnosis creates no resources, so there's nothing to clean up.
Google Cloud CLI 必须已安装,并已针对
{project_id}
完成身份验证。如果尚未安装,或者某次读取因
PERMISSION_DENIED
或 API 被禁用而失败,请加载 prerequisites.md。该文件涵盖
gcloud auth login
、API、只读角色、billing 和默认区域(
us-central1
)。诊断过程不会创建任何资源,因此无需清理。

Required context

所需上下文

Collect these values. Ask the user for any that are missing.
ValueSource
{project_id}
,
{project_number}
The agent's project, and the gateway's project if it's different
{location}
Region of the agent and Agent Gateway
{engine_id}
ID of the Agent Runtime agent
{gateway}
The Agent Gateway that carries the agent's traffic
Destination URLThe exact URL that the agent called
Error text and timeThe agent's logs or the user's report
{org_id}
Needed only for the optional principal access boundary check
If the agent and Agent Gateway are in different projects, use the agent's project number in principals. Use the gateway's project for the gateway, its authorization policies and extension, Access policies, and policy bindings. Run the IAP audit query in the gateway's project, and in the agent's project if it returns nothing.
Load topology-agent-gateway.md to confirm the topology, the principal format, the agent log query, and 403 errors during startup. If the agent doesn't run on Agent Runtime behind Agent Gateway, say that the topology isn't supported yet, and stop.
收集以下值。如果缺少任何值,请向用户询问。
值来源
{project_id}
、
{project_number}
Agent 所在项目;如果 Agent Gateway 位于不同项目,还需其项目
{location}
Agent 和 Agent Gateway 所在区域
{engine_id}
Agent Runtime Agent 的 ID
{gateway}
承载 Agent 流量的 Agent Gateway
目标 URLAgent 调用的准确 URL
错误文本和时间Agent 日志或用户提供的报告
{org_id}
仅可选的主体访问边界检查需要
如果 Agent 和 Agent Gateway 位于不同项目,请在 principal 中使用 Agent 项目的 project number。对于 Gateway、其授权策略和 extension、Access policies 以及 policy bindings,请使用 Gateway 所在项目。在 Gateway 所在项目中运行 IAP 审计查询;如果没有结果,则还要在 Agent 所在项目中运行。
加载 topology-agent-gateway.md,以确认拓扑结构、principal 格式、Agent 日志查询以及启动期间的 403 错误。如果 Agent 未运行在 Agent Gateway 后方的 Agent Runtime 上,请说明当前尚不支持该拓扑结构并停止。

Step 0: Classify the failure

第 0 步:对故障进行分类

Load audit-logs.md. Run the IAP decision query for the time of the failure, and match the entry to the request by principal, destination, and time. Use the first row that matches. An IAP entry outranks the error text, because the entry records the decision itself.
SignalMeaningNext
IAP entry with
granted: false
, not dry run
IAP denied the requestStep 1, then Steps 2-4
IAP entry with
dryRun: true
or
iamEnforcementMode: DRY_RUN
IAP logged the decision but didn't block the request. A 403 comes from the gateway or the destinationStep 1, then Steps 2-4: find the source of any 403, and whether IAP would deny the request once enforced
IAP entry with
granted: true
IAP allowed the requestStep 2, then the destination
No IAP entry, and the error is an IAM permission error on a model or Agent Platform API call, such as
aiplatform.endpoints.predict
denied
Not an IAP egress denialStop, and name
agent-platform-troubleshooting
No IAP entry for the requestData Access audit logs are off, or the gateway denied the request before IAPStep 1 (logs check), then Step 2
加载 audit-logs.md。针对故障发生时间运行 IAP 决策查询,并按 principal、目标和时间将条目与请求匹配。使用第一个匹配行。IAP 条目的优先级高于错误文本,因为该条目记录的是实际决策。
信号含义后续操作
IAP 条目中
granted: false
,且不是 dry run
IAP 拒绝了请求第 1 步,然后第 2-4 步
IAP 条目中
dryRun: true
或
iamEnforcementMode: DRY_RUN
IAP 记录了决策,但未阻止请求。403 来自 Gateway 或目标第 1 步,然后第 2-4 步:找出 403 的来源,并判断启用 enforcement 后 IAP 是否会拒绝该请求
IAP 条目中
granted: true
IAP 允许了请求第 2 步,然后检查目标
没有 IAP 条目,且错误是模型或 Agent Platform API 调用上的 IAM 权限错误,例如
aiplatform.endpoints.predict
被拒绝
这不是 IAP 出口拒绝停止,并指明
agent-platform-troubleshooting
请求没有 IAP 条目Data Access 审计日志未启用,或 Gateway 在 IAP 之前拒绝了请求第 1 步(日志检查),然后第 2 步

Step 1: Check telemetry

第 1 步:检查遥测数据

  1. Run the Data Access check in audit-logs.md. If the logs are off, explain that past requests weren't recorded, print the "Enable Data Access audit logs for IAP" template, and stop until the user reproduces the failure.
  2. From the matching entry, record
    principalSubject
    , the destination, the policy version, and the CEL evaluation details if the entry has them. Read each from the field names in the fields table of audit-logs.md, because entries use different layouts. The destination is
    metadata.destination
    when it's present, not
    authorizationInfo[].resource
    , which can name the gateway.
  3. If the entry has
    metadata.destination.unregistered
    , or
    audited_resource_name
    is
    unregisteredResource
    or ends in
    unregisteredEndpoint
    , the destination counts as unregistered: the hostname isn't registered, or, for an MCP server, the called tool isn't in its tool spec (see agent-registry.md). Keep that for Step 4.
  1. 运行 audit-logs.md 中的 Data Access 检查。如果日志未启用,请说明过去的请求未被记录,输出“Enable Data Access audit logs for IAP”模板,并停止,直到用户重现故障。
  2. 从匹配条目中记录
    principalSubject
    、目标、策略版本,以及条目中包含的 CEL evaluation details。请根据 audit-logs.md 的字段表中的字段名读取这些值,因为不同条目使用不同的布局。如果存在
    metadata.destination
    ,目标就是该字段,而不是可能指向 Gateway 的
    authorizationInfo[].resource
    。
  3. 如果条目包含
    metadata.destination.unregistered
    ,或者
    audited_resource_name
    为
    unregisteredResource
    或以
    unregisteredEndpoint
    结尾,则目标视为未注册:主机名未注册;或者对于 MCP server,被调用的工具不在其 tool spec 中(见 agent-registry.md)。请在第 4 步继续检查这一点。

Step 2: Check the gateway chain

第 2 步:检查网关链

Load gateway-setup.md. Read the gateway, its authorization policies, and its authorization extension.
  • If no authorization policy with
    policyProfile: REQUEST_AUTHZ
    lists the gateway in
    target.resources
    , the extension isn't executed and no IAP entry exists. That's the root cause.
  • Read the extension that the policy names, not the one you expect. Its
    service
    must be
    iap.googleapis.com
    . Record
    metadata.iapPolicyVersion
    ,
    metadata.iamEnforcementMode
    , and
    failOpen
    , because they decide which policies IAP evaluates, whether it blocks, and whether a request passes when the call to IAP fails or times out.
加载 gateway-setup.md。读取 Gateway、其授权策略和授权 extension。
  • 如果没有任何包含
    policyProfile: REQUEST_AUTHZ
    的授权策略在
    target.resources
    中列出该 Gateway,则 extension 不会执行,也不会生成 IAP 条目。这就是根因。
  • 读取策略所指定的 extension,而不是你预期的那一个。其
    service
    必须为
    iap.googleapis.com
    。记录
    metadata.iapPolicyVersion
    、
    metadata.iamEnforcementMode
    和
    failOpen
    ,因为这些字段决定 IAP 评估哪些策略、是否阻止请求,以及当对 IAP 的调用失败或超时时请求是否能够通过。

Step 3: Check policies and bindings

第 3 步:检查策略和绑定

Load policy-model.md, and use the section for the version from Step 2.
  • V2 (Access policies): read every policy bound to the project or the gateway, because a deny rule in any of them wins and a policy without a binding isn't enforced.
  • V1 (IAP allow policies): look for
    roles/iap.egressor
    granted to the agent's principal, or to a principal set that contains it, in the registry-wide policy and in the destination's own policy. A resource inherits the registry-wide grants, so a grant at either level applies.
  • Allowed but still denied: check principal access boundary policies (needs
    {org_id}
    ) and principal set membership (see topology-agent-gateway.md).
加载 policy-model.md,并使用第 2 步所确定版本对应的章节。
  • V2(Access policies): 读取绑定到项目或 Gateway 的每个策略,因为任一策略中的 deny 规则都会优先生效,而且没有 binding 的策略不会执行。
  • V1(IAP allow policies): 在 registry-wide policy 和目标自身的 policy 中,查找授予 Agent principal 或包含该 principal 的 principal set 的
    roles/iap.egressor
    。资源会继承 registry-wide grant,因此在任一层级授予权限都会生效。
  • 已允许但仍被拒绝: 检查主体访问边界策略(需要
    {org_id}
    )和 principal set 成员关系(见 topology-agent-gateway.md)。

Step 4: Check the rule match

第 4 步:检查规则匹配

Load cel-attributes.md and agent-registry.md.
  1. Principal: the rule lists the agent's exact
    principal://
    identifier, or a
    principalSet://
    that contains the agent: same trust domain, and the agent's project number or platform attribute, as listed in the principal set table of topology-agent-gateway.md. If
    principalSubject
    holds a SPIFFE ID, build the identifier as described there. A project ID in place of the project number, in either form, or a URN identifier, never matches.
  2. Destination: resolve the URL in Agent Registry.
    • Registered: the condition names the registry resource with
      ==
      or
      in
      , exactly as the audit entry's
      resource_id
      or the Agent Registry list output shows it.
    • Unregistered, Access policies (V2) only: the condition matches the host, the path, and the method (see cel-attributes.md). IAP allow policies (V1) can't allow an unregistered destination: register it, and grant
      roles/iap.egressor
      only if no registry-wide grant covers the agent.
  3. Condition: for V2, use the CEL evaluation details from the audit entry to find the clause that failed. If the entry has no evaluation details, compare the entry's
    resource_id
    or
    unregistered
    values with each literal in the condition yourself.
加载 cel-attributes.md 和 agent-registry.md。
  1. Principal: 规则应列出 Agent 的准确
    principal://
    identifier,或列出包含该 Agent 的
    principalSet://
    :具有相同的 trust domain,并包含 Agent 的 project number 或 platform attribute,具体如 topology-agent-gateway.md 的 principal set 表所列。如果
    principalSubject
    包含 SPIFFE ID,请按照该文件中的说明构建 identifier。无论采用哪种形式,以 project ID 代替 project number,或使用 URN identifier,都不会匹配。
  2. 目标: 在 Agent Registry 中解析 URL。
    • 已注册: 条件使用
      ==
      或
      in
      指定 registry resource,并且必须与审计条目的
      resource_id
      或 Agent Registry 列表输出所显示的内容完全一致。
    • 未注册,仅适用于 Access policies(V2): 条件必须匹配 host、path 和 method(见 cel-attributes.md)。IAP allow policies(V1)无法允许未注册目标:请注册该目标;仅当没有 registry-wide grant 覆盖该 Agent 时,才授予
      roles/iap.egressor
      。
  3. 条件: 对于 V2,使用审计条目中的 CEL evaluation details 找出失败的子句。如果条目不包含 evaluation details,请自行将条目中的
    resource_id
    或
    unregistered
    值与条件中的每个字面量进行比较。

Report

报告

Load only the remediation reference for the root cause:
Root causeLoad
Data Access audit logs are off, or an API is disabledremediation-setup.md
The authorization policy doesn't target the gateway, or calls the wrong extensionremediation-gateway.md
An Access policy (V2) rule doesn't match or a DENY rule matches, a binding is missing, or the organization policy constraint blocks the bindingremediation-access-policies.md
An IAP allow policy (V1) grant is missing, the destination or a hostname variant isn't registered, or an MCP tool is missing from the server's tool specremediation-registry.md
Use this template:
markdown
undefined
仅加载与根因对应的修复参考文件:
根因加载文件
Data Access 审计日志未启用,或 API 被禁用remediation-setup.md
授权策略未指向 Gateway,或调用了错误的 extensionremediation-gateway.md
Access policy(V2)规则不匹配或 DENY 规则匹配、缺少 binding,或组织策略约束阻止了 bindingremediation-access-policies.md
缺少 IAP allow policy(V1)grant、目标或主机名变体未注册,或服务器 tool spec 中缺少 MCP 工具remediation-registry.md
使用以下模板:
markdown
undefined

Root cause

Root cause

{root_cause_quoting_the_mismatched_value}
{root_cause_quoting_the_mismatched_value}

Fix (you run this)

Fix (you run this)

{why_smallest_change} {filled_in_template} Required role: {role}
{why_smallest_change} {filled_in_template} Required role: {role}

Wait time

Wait time

{wait_time}
{wait_time}

Verify

Verify

  • Reproduce the request.
  • A new IAP audit entry shows granted: true for {principal} and {destination_url}.
  • {read_command_that_confirms_the_change}

In dry run, IAP blocks nothing yet. Say so in the root cause, and say that its
logged denials are the requests that enforcement would block, because users
run dry run to find these requests before they enforce. If the user needs the
request to succeed and the policies wouldn't allow it once enforced, print
that fix under "Before you turn on enforcement". Verify it with a read
command, because a dry-run entry's `granted` value doesn't show whether a
policy allows the request.

If the evidence doesn't prove one root cause, list what you checked, what's
still unknown, and the next read command. Don't print a fix.
  • Reproduce the request.
  • A new IAP audit entry shows granted: true for {principal} and {destination_url}.
  • {read_command_that_confirms_the_change}

在 dry run 模式下,IAP 尚未阻止任何请求。应在根因中说明这一点,并说明其记录的拒绝正是启用 enforcement 后会阻止的请求,因为用户运行 dry run 是为了在启用 enforcement 前发现这些请求。如果用户需要请求成功,而相关策略在启用 enforcement 后仍不允许该请求,请将相应修复输出在“Before you turn on enforcement”下。使用只读命令进行验证,因为 dry-run 条目的 `granted` 值无法说明策略是否允许该请求。

如果证据无法证明存在唯一根因,请列出已检查的内容、仍然未知的内容以及下一条应执行的只读命令。不要输出修复方案。

Validation loop

验证循环

Before you send the report, check every item. If one fails, fix the report and check again.
  • No command or MCP tool call that you made changed a resource. Every fix is printed for the user.
  • The fix is additive. It keeps existing bindings, rules, and clauses.
  • If a DENY rule causes the denial, the report says that the block might be deliberate and offers any change to that rule only as an option for the rule's owner to approve.
  • The principal is the agent's exact
    principal://
    identifier, in the Agent Runtime format.
  • Registry resource names in conditions use the project ID, copied from the audit entry's
    resource_id
    or from Agent Registry list output.
  • A binding fix targets the project that contains Agent Gateway.
  • Hostnames match exactly, every variant is covered, and methods are in the request's case, such as
    'GET'
    .
  • The fix allows only the hosts, paths, and methods that the logs show the agent calling, and keeps restrictions that the rule already has, such as a method limit, because they're often deliberate. If the user asks to loosen a rule, say whether the logs show that it's needed. Never print a rule that's looser than the logs require.
  • V1 fixes use
    roles/iap.egressor
    , never
    roles/iap.tunnelResourceAccessor
    , which grants IAP TCP forwarding, not egress.
  • No fix turns off IAP or switches enforcement to dry run, because that removes egress protection for every agent behind the gateway.
  • Every placeholder is filled in from command output.
  • Commands printed for the user don't include
    CLOUDSDK_METRICS_REQUEST_ATTRIBUTION
    or the attribution
    User-Agent
    string, because the tag is only for commands you run yourself (see attribution.md).
发送报告前,请检查每一项。如果某项检查失败,请修正报告并重新检查。
  • 你执行的任何命令或 MCP 工具调用都未更改资源。所有修复方案均已输出给用户。
  • 修复采用增量方式。它会保留现有 bindings、rules 和 clauses。
  • 如果 DENY 规则导致拒绝,报告会指出该阻断可能是有意设置,并仅将该规则的任何更改作为需要规则所有者批准的可选方案。
  • principal 使用 Agent Runtime 格式的准确
    principal://
    identifier。
  • 条件中的 registry resource name 使用从审计条目的
    resource_id
    或 Agent Registry 列表输出中复制的 project ID。
  • binding 修复指向包含 Agent Gateway 的项目。
  • 主机名完全匹配,覆盖每个变体,method 与请求中的大小写一致,例如
    'GET'
    。
  • 修复仅允许日志显示 Agent 调用的 hosts、paths 和 methods,并保留规则已有的限制,例如 method limit,因为这些限制通常是有意设置。如果用户要求放宽规则,请说明日志是否表明确有必要。绝不输出比日志要求更宽松的规则。
  • V1 修复使用
    roles/iap.egressor
    ,绝不使用
    roles/iap.tunnelResourceAccessor
    ;后者授予的是 IAP TCP forwarding,而不是 egress。
  • 修复不会关闭 IAP,也不会将 enforcement 切换为 dry run,因为这样会移除 Gateway 后所有 Agent 的 egress 保护。
  • 每个 placeholder 都已根据命令输出填写。
  • 输出给用户的命令不包含
    CLOUDSDK_METRICS_REQUEST_ATTRIBUTION
    或归因
    User-Agent
    字符串,因为该 tag 仅用于你自己执行的命令(见 attribution.md)。