debugging-mwaa-workflow

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Debugging MWAA Workflows

调试 MWAA 工作流

AWS MCP server (optional but recommended): running the AWS CLI commands in this skill through the AWS MCP server gives sandboxed execution and audit logging. Every command here also works with the plain AWS CLI, so the skill does not require the MCP server or any MCP-only tools.

Diagnose and root-cause Amazon MWAA workflow failures, then report root cause, impact, and recommended remediation. Routes by flavor, then runs a shared 4-step diagnostic spine.

**AWS MCP server（可选但推荐）：**通过 AWS MCP server 运行此 skill 中的 AWS CLI 命令，可获得沙箱化执行和审计日志记录。此处每条命令也可使用普通 AWS CLI 运行，因此该 skill 不依赖 MCP server 或任何 MCP-only 工具。

诊断 Amazon MWAA 工作流故障并找出根因，然后报告根因、影响和建议的修复方案。先按环境类型分流，再运行共享的 4 步诊断主线。

Guardrail — where this skill's own files live (MCP vs local install)

防护规则——此 skill 自身文件的位置（MCP 与本地安装）

This skill can be loaded two ways, and they resolve the skill's own bundled files from different places. Determine how the skill was loaded before reading a reference:

Loaded through the AWS MCP
retrieve_skill
tool: The skill is not installed on the local filesystem. You MUST fetch each reference via
```
retrieve_skill
```
with the
```
file
```
parameter (e.g.
```
file="references/failure-catalog.md"
```
) and read the returned content. Do NOT
```
file_read
```
these paths locally — they do not exist on disk.
Installed locally (e.g.
```
.kiro/skills/debugging-mwaa-workflow/
```
or
```
~/.claude/skills/debugging-mwaa-workflow/
```
): Read the files from the local skill directory using relative paths.

This distinction applies only to the skill's own packaged files. User data and session artifacts are always read from and written to the user's working directory. Never fetch or write customer data through

retrieve_skill

此 skill 有两种加载方式，它们从不同位置解析此 skill 自带的文件。读取引用文件前，请先确定 skill 的加载方式：

**通过 AWS MCP
```
retrieve_skill
```
工具加载：**skill 未安装在本地文件系统中。你必须通过
```
retrieve_skill
```
并使用
```
file
```
参数获取每个引用文件（例如
```
file="references/failure-catalog.md"
```
），然后读取返回的内容。不要在本地对这些路径执行
```
file_read
```
——它们在磁盘上不存在。
本地安装（例如
```
.kiro/skills/debugging-mwaa-workflow/
```
或
```
~/.claude/skills/debugging-mwaa-workflow/
```
）：使用相对路径从本地 skill 目录读取文件。

此区别仅适用于此 skill 自身打包的文件。用户数据和会话工件始终从用户的工作目录读取，并写入用户的工作目录。切勿通过

retrieve_skill

获取或写入客户数据。

Step 0: Detect Flavor and Scope the Failure

步骤 0：检测环境类型和故障范围

Detect flavor

检测环境类型

An environment name resolvable via
```
aws mwaa get-environment
```
means the environment is Provisioned.
A
```
workflow/...
```
ARN or any
```
aws mwaa-serverless
```
context means the environment is Serverless. A bare run identifier does NOT indicate flavor — Provisioned DAG runs also have run ids.
If neither signal is present, ask: is the target MWAA Provisioned (Python DAG) or MWAA Serverless (YAML workflow)?

如果某个环境名称可通过
```
aws mwaa get-environment
```
解析，则该环境为 Provisioned。
如果存在
```
workflow/...
```
ARN 或任何
```
aws mwaa-serverless
```
上下文，则该环境为 Serverless。仅有 run identifier 并不能说明环境类型——Provisioned DAG 运行也有 run id。
如果两种信号都不存在，请询问：目标环境是 MWAA Provisioned（Python DAG） 还是 MWAA Serverless（YAML workflow）？

Route by complexity

按复杂度分流

Simple — a single named task or run failed with a clear exception. Jump to Step 2 for that task.
Standard — a run failed and the cause is unknown. Run the full Step 1 to Step 4 sweep.
Complex — intermittent or environment-wide (multiple DAGs, "worked yesterday", nothing appearing). Run the full sweep with emphasis on Step 3.

简单（Simple）——单个命名任务或运行失败，且有明确异常。直接跳至该任务的步骤 2。
标准（Standard）——运行失败，但原因未知。完整执行步骤 1 至步骤 4 的排查。
复杂（Complex）——间歇性故障或环境范围故障（多个 DAG、“昨天还可以运行”、没有任何内容出现）。执行完整排查，并重点关注步骤 3。

Step 1: Identify the Failure

步骤 1：识别故障

Provisioned: list failed DAG runs and task instances via

aws mwaa invoke-rest-api

(paths

/dags/{id}/dagRuns

and

/dags/{id}/dagRuns/{run_id}/taskInstances

). If

invoke-rest-api

errors (

RestApiClientException

), fall back to the Scheduler and DAGProcessing log groups. Get version and config from

aws mwaa get-environment

. See references/provisioned-diagnostics.md.

Serverless:

aws mwaa-serverless list-workflow-runs

, then

get-workflow-run

. Read

RunDetail.ErrorMessage

— an empty

TaskInstances

with a parser message is a definition error;

Workflow execution failed

with populated

TaskInstances

is a task-execution failure. See references/serverless-diagnostics.md.

**Provisioned：**通过

aws mwaa invoke-rest-api

列出失败的 DAG 运行和任务实例（路径为

/dags/{id}/dagRuns

和

/dags/{id}/dagRuns/{run_id}/taskInstances

）。如果

invoke-rest-api

报错（

RestApiClientException

），则回退到 Scheduler 和 DAGProcessing 日志组。通过

aws mwaa get-environment

获取版本和配置。参见 references/provisioned-diagnostics.md。

**Serverless：**先执行

aws mwaa-serverless list-workflow-runs

，再执行

get-workflow-run

。读取

RunDetail.ErrorMessage

——如果

TaskInstances

为空但包含 parser message，则属于 definition error；如果显示

Workflow execution failed

且

TaskInstances

非空，则属于 task-execution failure。参见 references/serverless-diagnostics.md。

Step 2: Get Error Details and Categorize

步骤 2：获取错误详情并进行分类

Pull the real exception past boilerplate:

Provisioned: read the Task log group first, then Worker/Scheduler/ DAGProcessing as the symptom directs.

Serverless:

list-task-instances

then

get-task-instance

to get each task's

LogStream

, then read that stream in CloudWatch.

Then categorize in priority order —

infra

, then

drift

, then

code-data

— using references/failure-catalog.md. The category determines the Step 3 checks.

越过通用样板信息，提取真正的异常：

**Provisioned：**先读取 Task 日志组，然后根据症状检查 Worker/Scheduler/DAGProcessing。

**Serverless：**先执行

list-task-instances

，再执行

get-task-instance

获取每个任务的

LogStream

，然后在 CloudWatch 中读取该流。

然后按优先级分类——先判断

infra

，再判断

drift

，最后判断

code-data

——并使用 references/failure-catalog.md。类别决定步骤 3 的检查项。

Step 3: Check Context (Why It Happened)

步骤 3：检查上下文（故障为何发生）

Run the context checks for the matched category from references/failure-catalog.md. Do not stop at the surface exception: a SIGKILL is an OOM story, a fresh import error on unchanged code is a drift story, a sensor timeout is an upstream-health story.

根据 references/failure-catalog.md 中匹配类别的上下文检查项执行排查。不要止步于表面异常：SIGKILL 意味着 OOM；代码未改动却新出现 import error 意味着 drift；sensor timeout 则反映上游健康状况。

Step 4: Provide Actionable Output

步骤 4：提供可执行的输出

Report in this exact structure:

Root Cause: <one-line diagnosis with the evidence that proves it>
Impact: <what failed, which runs, blast radius>
Immediate Fix: <the smallest change that unblocks>
Prevention: <the change that stops recurrence>
Commands: <exact read-only commands run, plus remediation commands for the user to run>

Run only read-only operations. Present state-mutating remediation (clear/rerun/backfill for Provisioned; start-workflow-run or fix-and-redeploy for Serverless) as commands for the user to run, with the impact stated. Never execute them autonomously (production safety).

For the fix-and-redeploy path, use

authoring-mwaa-workflow

to regenerate a compliant artifact.

严格按以下结构报告：

Root Cause: <基于证据的一行根因诊断>
Impact: <失败内容、受影响的运行及影响范围>
Immediate Fix: <解除阻塞所需的最小变更>
Prevention: <防止再次发生的变更>
Commands: <实际执行的只读命令，以及供用户运行的补救命令>

仅运行只读操作。对于会改变状态的补救措施（Provisioned 的 clear/rerun/backfill；Serverless 的 start-workflow-run 或 fix-and-redeploy），应将其作为命令提供给用户运行，并说明影响。绝不自动执行这些操作（生产安全）。

对于 fix-and-redeploy 路径，使用

authoring-mwaa-workflow

重新生成符合要求的 artifact。

Gotchas

注意事项

Serverless has no Airflow web UI, no REST API, and no CLI token. Do not attempt
```
create-web-login-token
```
,
```
invoke-rest-api
```
, or any Airflow REST path for Serverless.
For Provisioned, always use
```
aws mwaa invoke-rest-api
```
(not
```
create-web-login-token
```
+ curl).
```
invoke-rest-api
```
reaches VPC-only web servers without network access.
```
GetWorkflowRun.RunDetail.ErrorMessage
```
distinguishes a definition error (empty
```
TaskInstances
```
) from a task-execution failure (
```
Workflow execution failed
```
, populated
```
TaskInstances
```
). Read it before pulling task logs.
The Serverless log group defaults to
```
/aws/mwaa-serverless/{workflow-id}/
```
but can be a custom group; confirm via
```
get-workflow
```
```
LoggingConfiguration
```
before assuming the path.
A DAG not appearing has several causes — an import/parse error, the scheduler scan interval (
```
scheduler.dag_dir_list_interval
```
, or
```
dag_processor.refresh_interval
```
on Airflow 3.x) not yet elapsed, a
```
dag_id
```
collision, or S3-sync delay — and is rarely a broken DAG. Check
```
GET /importErrors
```
and
```
GET /dags/{dag_id}
```
via invoke-rest-api (and the DAGProcessing logs); see the failure catalog's "DAG not appearing in the UI" checklist before concluding the code is wrong.
A worker SIGKILL is an OOM signal. Recommend moving work to Glue/EMR/Lambda; scaling workers alone does not fix per-task memory pressure.
MWAA re-resolves dependencies on environment update, so an unchanged DAG can start failing on import with no code change. Treat no-code-change import failures as drift.
Serverless
```
PythonOperator
```
/
```
BashOperator
```
tasks run custom code from a
```
--code
```
package. A run that fails to extract the package or hits
```
ModuleNotFoundError
```
is a packaging problem (wrong-platform wheel, missing dep, bad layout), not a YAML definition error. See the failure catalog's serverless custom-code section.

Serverless 没有 Airflow Web UI、REST API 或 CLI token。不要尝试对 Serverless 使用
```
create-web-login-token
```
、
```
invoke-rest-api
```
或任何 Airflow REST 路径。
对于 Provisioned，始终使用
```
aws mwaa invoke-rest-api
```
（不要使用
```
create-web-login-token
```
+ curl）。
```
invoke-rest-api
```
无需网络访问即可访问仅在 VPC 内开放的 Web 服务器。
```
GetWorkflowRun.RunDetail.ErrorMessage
```
可区分 definition error（
```
TaskInstances
```
为空）与 task-execution failure（显示
```
Workflow execution failed
```
，且
```
TaskInstances
```
非空）。读取任务日志前先检查它。
Serverless 日志组默认路径为
```
/aws/mwaa-serverless/{workflow-id}/
```
，但也可以使用自定义日志组；不要想当然地采用该路径，应先通过
```
get-workflow
```
的
```
LoggingConfiguration
```
进行确认。
DAG 未出现有多种原因——import/parse error、调度器扫描间隔（
```
scheduler.dag_dir_list_interval
```
，或 Airflow 3.x 中的
```
dag_processor.refresh_interval
```
）尚未到期、
```
dag_id
```
冲突或 S3 同步延迟——很少是 DAG 本身损坏。通过
```
invoke-rest-api
```
检查
```
GET /importErrors
```
和
```
GET /dags/{dag_id}
```
（同时检查 DAGProcessing 日志）；在断定代码有问题之前，先查看 failure catalog 中“DAG 未出现在 UI 中”的检查清单。
Worker SIGKILL 是 OOM 信号。建议将工作转移到 Glue/EMR/Lambda；仅扩展 worker 无法解决单任务内存压力。
MWAA 会在更新环境时重新解析依赖项，因此即使 DAG 未发生变化，也可能开始出现 import error。将代码未变化却出现 import failure 的问题视为 drift。
Serverless
```
PythonOperator
```
/
```
BashOperator
```
任务通过
```
--code
```
package 运行自定义代码。如果某次运行无法解压 package，或出现
```
ModuleNotFoundError
```
，则这是 packaging 问题（平台错误的 wheel、缺少依赖项或目录结构错误），而不是 YAML definition error。参见 failure catalog 中的 Serverless custom-code 部分。

Troubleshooting

故障排除

Error	Cause	Fix
`RestApiClientException` (Provisioned)	Mis-scoped execution role or service error	Fall back to Scheduler/DAGProcessing log groups
`ResourceNotFoundException` on get-workflow-run	Wrong workflow ARN or run id	Re-list with list-workflow-runs
Task log stream empty (Serverless)	Wrong log group assumed	Read `LoggingConfiguration` from get-workflow
No task logs but run FAILED	Definition/parse error	Read `RunDetail.ErrorMessage` ; fix the YAML

错误	原因	修复方式
`RestApiClientException` （Provisioned）	execution role 范围配置错误或服务错误	回退到 Scheduler/DAGProcessing 日志组
对 get-workflow-run 抛出 `ResourceNotFoundException`	workflow ARN 或 run id 错误	使用 list-workflow-runs 重新列出
Task log stream 为空（Serverless）	假设了错误的日志组	从 get-workflow 读取 `LoggingConfiguration`
无任务日志但运行状态为 FAILED	definition/parse error	读取 `RunDetail.ErrorMessage` ；修复 YAML

References

参考资料

references/provisioned-diagnostics.md — Provisioned data sources and read-only commands
references/serverless-diagnostics.md — Serverless API, log group, failure classes
references/failure-catalog.md — category-keyed context checks and remediation

references/provisioned-diagnostics.md——Provisioned 数据源和只读命令
references/serverless-diagnostics.md——Serverless API、日志组和故障类别
references/failure-catalog.md——按类别组织的上下文检查项和补救措施

Security Considerations

安全注意事项

Read-only by default: diagnosis uses only read/list/describe calls. Remediation (clear/rerun/backfill, IAM or key-policy changes) is presented as commands for the user to run, never executed autonomously.
Least-privilege IAM: when an
```
AccessDenied
```
is a genuine permission gap, recommend the minimal
```
Action
```
/
```
Resource
```
from the error — never a wildcard; distinguish it from a nonexistent-resource typo (do not broaden IAM then).
Cross-account: KMS key-policy / assume-role changes are human-gated and coordinated with the resource owner.
No secret exposure: do not surface credentials or connection strings from logs or API responses in the diagnosis output.

**默认只读：**诊断仅使用 read/list/describe 调用。补救措施（clear/rerun/backfill、IAM 或 key-policy 更改）以命令形式提供给用户运行，绝不自动执行。
**最小权限 IAM：**当
```
AccessDenied
```
确实源于权限缺失时，建议使用错误信息中所需的最小
```
Action
```
/
```
Resource
```
——绝不使用通配符；同时将其与不存在的资源拼写错误区分开来（不要因此扩大 IAM 权限）。
**跨账号：**KMS key-policy / assume-role 更改需要人工把关，并与资源所有者协调。
**不暴露机密：**诊断输出中不得呈现日志或 API 响应里的凭据或连接字符串。