API Gateway Troubleshooting

Use this skill to find out why requests to a deployed Google Cloud API Gateway fail and how to fix them. Work only from three sources of evidence:

Public Google Cloud documentation, linked from this skill.
The user's own project, read with the user's own credentials.
Files and output that the user provides, such as an OpenAPI document, an error response, or a pasted log entry.

Scope

This skill covers only Google Cloud API Gateway. If the question is about Apigee, Cloud Endpoints, or GKE Gateway API, reply that this skill covers only Google Cloud API Gateway, and don't run its steps or commands.

Usage Attribution

MUST READ: Before executing any

gcloud

, REST API, or MCP command, you MUST read and follow the instructions in Usage Attribution.

Ground rules

Rule	Why
Stay read-only until the user approves a change. While you diagnose, run only `list` , `describe` , `get-iam-policy` , and `logging read` commands, plus `nslookup` and `curl` `GET` requests to a path that's safe to call.	Gateways serve production traffic, and the quota metrics and limits of the most recently created API config apply to every gateway of the API.
Before any change, show the exact command, what it changes, and how to undo it, then wait for an explicit yes. Changes include IAM grants, `api-configs create` , `gateways update` , Cloud Run ingress changes, and `gcloud services enable` .	The user owns the project and the change.
Never fix a `401` or `403` by making a backend public ( `--allow-unauthenticated` , `--no-invoker-iam-check` , `allUsers` ) or by granting a basic role such as Editor.	Grant the gateway's service account the invoker role on the one backend instead, which keeps least privilege.
Keep every log read bounded: filter on the gateway, set `--freshness` or a timestamp window, and set `--limit` .	Unbounded reads are slow and pull in caller data that the diagnosis doesn't need.
Redact API keys, bearer tokens, and JWTs before you show log entries, commands, or files.	Request logs and pasted output can contain credentials.
Base every limit, default, and error meaning on a linked public doc or on output from the user's project. If neither explains the problem, say so and escalate.	Guessing about undocumented behavior leads to wrong fixes.
Use the user's existing `gcloud` login. Never ask for passwords, API key values, or service account key files.	Credentials don't belong in a chat transcript.
For Preview features (streaming, MCP, model routing), check the release notes before answering.	Preview behavior changes between releases.

Prerequisites

Google Cloud CLI. Use a recent
```
gcloud
```
(the commands here were checked with version 591.0.0); see Install the gcloud CLI. Don't install or update software without the user's approval. If
```
gcloud
```
isn't available or the user says not to run commands, work in evidence mode: give the user the read-only commands to run, and diagnose from the output, response bodies, and log entries they paste.
Authentication. Use the user's existing
```
gcloud auth login
```
session; this skill doesn't need application-default credentials.
Project. Ask which project hosts the gateway and pass
```
--project={project_id}
```
on every command. Don't run
```
gcloud config set
```
, because it changes the user's environment. This skill troubleshoots existing gateways and never creates projects.
Billing. The project needs an active billing account even when usage stays in the free tier; see API Gateway pricing.
APIs. API Gateway requires
```
apigateway.googleapis.com
```
,
```
servicemanagement.googleapis.com
```
, and
```
servicecontrol.googleapis.com
```
. A gateway can keep responding while one of them is missing, and its logs then stay empty, so check with
```
gcloud services list --enabled --project={project_id}
```
. Enabling a missing API is a change that needs approval.
IAM roles for diagnosis (read-only):
```
roles/apigateway.viewer
```
,
```
roles/logging.viewer
```
,
```
roles/serviceusage.serviceUsageViewer
```
, and
```
roles/iam.serviceAccountViewer
```
(to check the gateway's service account). For Cloud Run backends, add
```
roles/run.viewer
```
; to trace a custom domain through its load balancer, add
```
roles/compute.viewer
```
; to check an API key's project,
```
roles/serviceusage.serviceUsageViewer
```
is also needed there.
IAM roles for fixes (only after approval):
```
roles/apigateway.admin
```
;
```
roles/iam.serviceAccountUser
```
on the gateway's service account, because creating or updating an API config or a gateway requires
```
iam.serviceAccounts.actAs
```
;
```
roles/run.admin
```
to change a Cloud Run service's IAM policy or ingress;
```
roles/serviceusage.serviceUsageAdmin
```
to enable an API or a managed service, plus
```
roles/servicemanagement.serviceConsumer
```
on the API's managed service to enable it in another project; and
```
roles/iam.serviceAccountAdmin
```
to re-enable a disabled service account.
Region. Gateways are regional. Use the region that
```
gateways list
```
reports. When a fix needs a new gateway, create it in the original gateway's region, or in
```
us-central1
```
if the user has no preference; see API Gateway deployment model and Cloud locations.

Workflow

Copy this checklist into your notes and tick it off as you go:

- [ ] 1. Context collected
- [ ] 2. Gateway identified by defaultHostname
- [ ] 3. Expected or transient states ruled out
- [ ] 4. Gateway log entry read (bounded query)
- [ ] 5. One branch chosen from the classification table
- [ ] 6. Root cause stated with redacted evidence and a doc link
- [ ] 7. Fix proposed, approved, and applied (or handed to the user)
- [ ] 8. Fix verified
- [ ] 9. Escalated, if the evidence points to Google Cloud

1. Collect context

Extract these from the conversation first, and ask only for what's missing:

Needed	Why
`{project_id}` of the gateway	Every command is scoped to it.
Failing URL (host and path) or `{gateway_id}` and `{region}`	Identifies the gateway.
HTTP method, status code, and response body	Separates gateway errors from backend errors.
Approximate time of the failure, in UTC	Bounds the log query.
What changed recently (new API config, IAM, backend deploy, DNS)	Recent changes are the first place to look.
Authentication used by the caller (API key, JWT, none)	Selects the auth branch.

If the project or the host is unknown, stop and ask before you run any

gcloud

command. Don't guess project IDs, and don't use the active

gcloud

project instead.

2. Identify the gateway

List the project's gateways:

bash

gcloud api-gateway gateways list --project={project_id} \
  --format="table(name.basename():label=GATEWAY,name.segment(3):label=REGION,defaultHostname,state,apiConfig)"

Match the request host against

defaultHostname

. Don't decide from the URL suffix: a gateway's hostname can use either of two

gateway.dev

formats or, for older streaming and model-routing gateways, a

run.app

format. Don't guess the owning project from numbers in the hostname either; list the gateways in the project the user names. A custom domain points at a load balancer whose serverless NEG targets the gateway. See references/triage-and-logs.md for the hostname formats and how to trace a custom domain.

If no gateway in the project matches, confirm the product before going on (see Scope).

Then read the gateway's details, including its API config and streaming mode:

bash

gcloud api-gateway gateways describe {gateway_id} \
  --location={region} --project={project_id}

3. Rule out expected and transient states

Gateway
state
isn't
ACTIVE
:
```
CREATING
```
or
```
UPDATING
```
means an operation is still running, so wait and check again.
```
FAILED
```
means creation failed; read the error of the failed operation.
```
DELETING
```
means the gateway is being removed.
API config just created: creating a complex API config can take up to ten minutes, and only one config can be created for an API at a time.
First requests after an idle period: API Gateway has cold-start latency.
Google Cloud incident: check the Service Health dashboard.

4. Read the gateway log entry

Use a bounded query that prints only the fields you need. Replace

{status_code}

with the status the caller saw:

bash

gcloud logging read 'resource.type="apigateway.googleapis.com/Gateway" AND resource.labels.gateway_id="{gateway_id}" AND resource.labels.location="{region}" AND httpRequest.status={status_code}' \
  --project={project_id} --freshness=1h --limit=20 \
  --format="json(timestamp,httpRequest.status,httpRequest.latency,httpRequest.requestUrl,jsonPayload.responseDetails,jsonPayload.apiConfig,jsonPayload.apiMethod)"

If you know the failure time, replace

--freshness

with a

timestamp>="{start_utc}" AND timestamp<="{end_utc}"

filter that spans about 30 minutes around it. Read

httpRequest.status

httpRequest.requestUrl

jsonPayload.responseDetails

jsonPayload.apiConfig

, and

jsonPayload.apiMethod

. Keep the

--format

field list, because a full entry can contain the caller's credentials, and redact credentials before you quote any field, including a

key

query parameter in

httpRequest.requestUrl

To compare API configs, compare only the

apis/{api_id}/configs/{config_id}

part: the same config can appear as a full resource name (

//apigateway.googleapis.com/projects/...

) and with either the project number or the project ID.

jsonPayload.responseDetails

decides the branch:

via_upstream

means the backend produced the response; any other value means the gateway produced it. The troubleshooting page explains

via_upstream

and

service_control_quota_error

, and the model routing page explains the

model_router_*

values that model-routing gateways log. Quote any other value as-is, and don't explain how the gateway works internally.

5. Classify and follow one branch

Pick the first row that matches, and load only the reference file it names.

What you see	Likely cause	Reference
`401` or `403` with `via_upstream`	The backend rejected the gateway: invoker role, audience, or deleted service account	auth-errors.md
A `4xx` produced by the gateway whose body names a credential problem (for example `UNAUTHENTICATED` )	The API key or JWT was rejected; quote the body	auth-errors.md
`500` with no gateway log entry, after log exclusions and a wrong scope are ruled out	The gateway's service account may be deleted or disabled	auth-errors.md
`500` with `service_control_quota_error`	Quota metric missing from the newest API config	config-deployment-errors.md
`429` with `service_control_quota_error`	The caller exceeded the API's quota	limits-and-escalation.md
`404` , `502` , `503` , or `504` , an oversized request, or a timeout	Routing, backend errors, size limits, or the deadline. The docs don't name the status codes for size or deadline failures, so confirm with `responseDetails` and the body.	backend-routing-and-timeouts.md
CORS preflight fails or latency is high	CORS pass-through or slow backend	backend-routing-and-timeouts.md
`api-configs create` fails with an error about the OpenAPI document, such as an unsupported extension or an invalid path, parameter, server, or security requirement	A spec problem	Hand off to `api-gateway-openapi-spec-authoring` ; if it isn't installed, see OpenAPI pre-flight check
`api-configs create` fails with a service account, concurrency, or limit error, or `gateways create` or `gateways update` fails	Service account, concurrent creates, limits, or the backend deadline	config-deployment-errors.md
`400` "The request was malformed" from an old TLS client, TLS scan findings, custom domain, or no log entry at all	TLS enforcement, DNS, load balancer, or wrong host	networking-domains-tls.md
Errors on several gateways, or on an unchanged config, with healthy backends	Possible Google Cloud issue	limits-and-escalation.md

If the request also needs log queries or backend inspection commands, use triage-and-logs.md.

6. State the root cause

Give the cause, the evidence (redacted, quoted verbatim from logs or command output), and the public doc section that explains it. If the evidence fits more than one cause, list at most three, say what would tell them apart, and collect that evidence before proposing a fix.

7. Propose the fix

Show the exact command, the resource it changes, its effect on live traffic, and how to undo it. Wait for approval, then run it, or hand it to the user if they prefer. Keep these rules in mind:

To change the OpenAPI document, create a new API config and deploy it with
```
gcloud api-gateway gateways update
```
. Before you create the config, check quota metrics with the
```
api-gateway-backend-security-and-rate-limiting
```
skill, because the newest API config's quota metrics apply to every gateway of the API.
Before you create a new API config, check the OpenAPI document with the linter (see OpenAPI pre-flight check).

8. Verify

Read the logs again for new entries after the change, or repeat the failing request. Repeat a request yourself only if it's safe to repeat, such as a

GET

; otherwise ask the user to repeat it, because a repeated

POST

can change data in their backend. After

gateways update

, wait until the gateway's

state

ACTIVE

before you test.

9. Escalate when the evidence points to Google Cloud

Escalate to Cloud Customer Care when the service account is active, the required APIs are enabled, and no log exclusion or wrong scope explains it, but requests get

errors with no gateway log entry; when several gateways fail at once without a config change; when errors are intermittent on an unchanged config while the backend logs stay clean; when control-plane operations keep failing with internal errors; or when Service Health reports an incident. Draft the case text with the evidence bundle in limits-and-escalation.md. The user files the case.

OpenAPI pre-flight check

Run the linter on every OpenAPI file before creating an API config. Run it; don't read its source. If you don't have the file, give the user the command to run:

bash

python3 scripts/lint_openapi_for_gateway.py {spec_path} [{spec_path} ...]

It reads the files only, makes no network calls, and prints JSON findings that each link to the public doc for the limitation. Exit code

means no errors,

means at least one

ERROR

finding, and

means a usage, parse, or dependency error. The linter needs Python 3.8 or later. YAML input needs PyYAML; ask the user before you install it with

pip install "PyYAML>=6.0,<7.0"

. JSON input doesn't. Fix the

ERROR

findings, and then create the API config. If

api-configs create

still fails on the OpenAPI document, hand off to

api-gateway-openapi-spec-authoring

, or use the error message if that skill isn't installed. Show

WARNING

findings to the user and explain them.

Output format

Report findings in this form:

**Problem:** <one sentence in the user's terms>
**Failed hop:** <gateway or backend, and the evidence for it>
**Evidence:** <redacted log fields or command output, verbatim>
**Root cause:** <one or two sentences> (<public doc link>)
**Fix:** <exact command or spec change; approval needed: yes/no>
**Verify:** <how to confirm the fix>
**Open questions:** <anything you couldn't confirm>

Related skills

Need	Skill
Write or change API key, JWT, backend authentication, or quota configuration	`api-gateway-backend-security-and-rate-limiting`
Write, migrate, or fix an OpenAPI spec, including OpenAPI errors from `api-configs create`	`api-gateway-openapi-spec-authoring`
MCP `/mcp` JSON-RPC errors on an MCP-enabled gateway	`api-gateway-mcp-server`
Deep IAM analysis, such as deny policies or role discovery	`iam-helper-for-troubleshooting`
Build a load balancer, Cloud Armor policy, or custom domain front end	`google-cloud-global-frontend-configuration`
Deploy or reconfigure a Cloud Run backend	`cloud-run-basics`

Reference files

references/attribution.md: usage attribution rules for every command you run.
references/triage-and-logs.md: hostname formats, log queries, log fields, metrics, and backend inspection commands.
references/auth-errors.md:
```
401
```
and
```
403
```
on either hop, API keys, JWTs, and the gateway service account.
references/backend-routing-and-timeouts.md:
```
404
```
, size limits, backend
```
5xx
```
,
```
504
```
and deadlines, CORS, and streaming.
references/config-deployment-errors.md: API config and gateway create, update, and delete failures other than OpenAPI errors, and quota metric conflicts.
references/networking-domains-tls.md: custom domains, load balancers, TLS, and requests that never reach the gateway.
references/limits-and-escalation.md: product limits, quota increases, unsupported features, and the Cloud Customer Care evidence bundle.
scripts/lint_openapi_for_gateway.py: offline linter for documented OpenAPI limitations.

api-gateway-troubleshooting

NPX Install

Tags

SKILL.md Content

API Gateway Troubleshooting

Scope

Usage Attribution

Ground rules

Prerequisites

Workflow

1. Collect context

2. Identify the gateway

3. Rule out expected and transient states

4. Read the gateway log entry

5. Classify and follow one branch

6. State the root cause

7. Propose the fix

8. Verify

9. Escalate when the evidence points to Google Cloud

OpenAPI pre-flight check

Output format

Related skills

Reference files