querying-aws-cloudwatch
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseQuery AWS CloudWatch System Tables
查询AWS CloudWatch系统表
Overview
概述
Works best with the AWS MCP server for sandboxed execution and audit logging. All commands below use the AWS CLI and work in any environment with configured AWS credentials.
The CloudWatch Logs S3 Tables integration exports log data as Apache Iceberg tables in the AWS-managed table bucket. This enables SQL analysis via Amazon Athena and correlation of log data with non-CloudWatch data (S3 metadata, business tables, etc.). Available at no additional storage charge beyond CloudWatch ingestion pricing.
aws-cloudwatch搭配AWS MCP服务器使用效果最佳,可实现沙箱执行和审计日志记录。以下所有命令均使用AWS CLI,可在已配置AWS凭证的任何环境中运行。
CloudWatch Logs S3 Tables集成会将日志数据导出为AWS托管的表存储桶中的Apache Iceberg表。这支持通过Amazon Athena进行SQL分析,并将日志数据与非CloudWatch数据(S3元数据、业务表等)关联。除CloudWatch摄入费用外,无额外存储费用。
aws-cloudwatchDecision Tree
决策树
| User intent | Use this skill? | Alternative |
|---|---|---|
| Run SQL across large volumes of log data | Yes | — |
| Correlate logs with S3 metadata or other tables | Yes — join across catalogs | — |
| Quick log search / pattern matching | No | CloudWatch Logs Insights (faster for ad-hoc) |
| Real-time log streaming/tailing | No | CloudWatch Logs console or |
| Set up alarms on log patterns | No | CloudWatch Metric Filters / Alarms |
| Query historical logs before integration was enabled | No | CloudWatch Logs (no backfill in S3 Tables) |
| 用户意图 | 是否使用此技能? | 替代方案 |
|---|---|---|
| 对大量日志数据运行SQL | 是 | — |
| 将日志与S3元数据或其他表关联 | 是 — 跨目录关联 | — |
| 快速日志搜索/模式匹配 | 否 | CloudWatch Logs Insights(临时查询速度更快) |
| 实时日志流/跟踪 | 否 | CloudWatch Logs控制台或 |
| 针对日志模式设置告警 | 否 | CloudWatch指标过滤器/告警 |
| 查询集成启用前的历史日志 | 否 | CloudWatch Logs(S3 Tables不支持回填) |
Supported Data Sources
支持的数据源
The following data sources are available through the S3 Tables integration. Each data source has a namespace pattern used in SQL queries. Not all AWS vended data sources may be available in all Regions; check the CloudWatch console Data Sources tab for current availability.
| Data Source | Namespace pattern | Common use case |
|---|---|---|
| VPC Flow Logs | | Network traffic analysis, rejected connections |
| WAF Logs | | Blocked requests, rule hit analysis |
| CloudFront Access Logs | | CDN traffic patterns, error rates |
| Route 53 Resolver Query Logs | | DNS query analysis |
| Network Firewall Logs | | Firewall rule hits, dropped traffic |
| EKS Audit Logs | | Kubernetes API audit trail |
| Verified Access Logs | | Zero-trust access decisions |
| SES Mail Logs | | Email delivery/bounce tracking |
| VPC Lattice Access Logs | | Service-to-service access patterns |
| Step Functions Logs | | Workflow execution debugging |
| Global Accelerator Flow Logs | | Global network traffic |
| NLB Access Logs | | Load balancer request tracing |
| Shield Logs | | DDoS mitigation events |
| Cognito Logs | | Auth/identity operations |
| ElastiCache Logs | | Redis slow log, engine log |
| SageMaker Logs | | ML training/inference events |
| WorkMail Audit Logs | | Email security/compliance |
| Bedrock Agent Logs | | AI agent invocations |
| Client VPN Logs | | VPN connection tracking |
| Entity Resolution Logs | | Record matching operations |
| MediaPackage Access Logs | | Streaming delivery metrics |
| MediaTailor Logs | | Ad insertion events |
| Transfer Family Logs | | SFTP/FTPS file transfer tracking |
| Site-to-Site VPN Logs | | VPN tunnel diagnostics |
Note: This table lists the 24 most commonly queried data sources. The integration supports 43+ AWS vended data sources in total. Useon thelist-namespacesbucket to discover all available data sources in your account. Namespace patterns follow the conventionaws-cloudwatch.<service>__<type>
以下数据源可通过S3 Tables集成获取。每个数据源都有一个用于SQL查询的命名空间模式。并非所有AWS官方数据源在所有区域都可用;请查看CloudWatch控制台的“数据源”选项卡了解当前可用情况。
| 数据源 | 命名空间模式 | 常见用例 |
|---|---|---|
| VPC Flow Logs | | 网络流量分析、拒绝连接情况 |
| WAF Logs | | 拦截请求分析、规则命中情况 |
| CloudFront Access Logs | | CDN流量模式、错误率分析 |
| Route 53 Resolver Query Logs | | DNS查询分析 |
| Network Firewall Logs | | 防火墙规则命中情况、丢弃流量分析 |
| EKS Audit Logs | | Kubernetes API审计追踪 |
| Verified Access Logs | | 零信任访问决策分析 |
| SES Mail Logs | | 邮件投递/退回跟踪 |
| VPC Lattice Access Logs | | 服务间访问模式分析 |
| Step Functions Logs | | 工作流执行调试 |
| Global Accelerator Flow Logs | | 全球网络流量分析 |
| NLB Access Logs | | 负载均衡器请求追踪 |
| Shield Logs | | DDoS缓解事件分析 |
| Cognito Logs | | 身份验证/身份操作分析 |
| ElastiCache Logs | | Redis慢日志、引擎日志分析 |
| SageMaker Logs | | 机器学习训练/推理事件分析 |
| WorkMail Audit Logs | | 邮件安全/合规审计 |
| Bedrock Agent Logs | | AI Agent调用分析 |
| Client VPN Logs | | VPN连接跟踪 |
| Entity Resolution Logs | | 记录匹配操作分析 |
| MediaPackage Access Logs | | 流媒体交付指标分析 |
| MediaTailor Logs | | 广告插入事件分析 |
| Transfer Family Logs | | SFTP/FTPS文件传输跟踪 |
| Site-to-Site VPN Logs | | VPN隧道诊断 |
注意:此表格列出了24种最常查询的数据源。该集成总共支持43余种AWS官方数据源。在存储桶上使用aws-cloudwatch命令可发现您账户中所有可用的数据源。命名空间模式遵循list-namespaces的约定。<service>__<type>
Common Tasks
常见任务
1. Check If Configured
1. 检查是否已配置
bash
undefinedbash
undefinedCheck if the aws-cloudwatch table bucket exists
检查aws-cloudwatch表存储桶是否存在
aws s3tables list-table-buckets --region <REGION>
--query "tableBuckets[?name=='aws-cloudwatch']"
--query "tableBuckets[?name=='aws-cloudwatch']"
- Empty result → integration not enabled. Guide user through setup.
- Bucket exists but no namespaces → integration enabled but no log data yet (only captures events *after* association).
List available tables:
```bash
aws s3tables list-namespaces --table-bucket-arn arn:aws:s3tables:<REGION>:<ACCOUNT>:bucket/aws-cloudwatch --region <REGION>
aws s3tables list-tables --table-bucket-arn arn:aws:s3tables:<REGION>:<ACCOUNT>:bucket/aws-cloudwatch --namespace <NAMESPACE> --region <REGION>aws s3tables list-table-buckets --region <REGION>
--query "tableBuckets[?name=='aws-cloudwatch']"
--query "tableBuckets[?name=='aws-cloudwatch']"
- 空结果 → 未启用集成。引导用户完成设置。
- 存储桶存在但无命名空间 → 已启用集成,但尚无日志数据(仅捕获关联*之后*的事件)。
列出可用表:
```bash
aws s3tables list-namespaces --table-bucket-arn arn:aws:s3tables:<REGION>:<ACCOUNT>:bucket/aws-cloudwatch --region <REGION>
aws s3tables list-tables --table-bucket-arn arn:aws:s3tables:<REGION>:<ACCOUNT>:bucket/aws-cloudwatch --namespace <NAMESPACE> --region <REGION>2. Enable / Configure
2. 启用/配置
Create integration:
bash
aws observabilityadmin create-s3-table-integration \
--region <REGION> \
--encryption '{"SseAlgorithm": "aws:kms", "KmsKeyArn": "<KMS_KEY_ARN>"}' \
--role-arn <SERVICE_ROLE_ARN>Associate a specific data source (recommended):
bash
aws logs associate-source-to-s3-table-integration \
--region <REGION> \
--integration-arn <INTEGRATION_ARN> \
--data-source '{"name": "<source-name>", "type": "<source-type>"}'Associate all data sources (wildcard):
⚠️ Warning: Wildcard association delivers all current and future data sources to S3 Tables. Use specific associations for tighter control over what log data lands in queryable tables.
bash
aws logs associate-source-to-s3-table-integration \
--region <REGION> \
--integration-arn <INTEGRATION_ARN> \
--data-source '{"name": "*", "type": "*"}'For IAM requirements (service role trust policy, permissions policy, condition keys), see Security Considerations below.
创建集成:
bash
aws observabilityadmin create-s3-table-integration \
--region <REGION> \
--encryption '{"SseAlgorithm": "aws:kms", "KmsKeyArn": "<KMS_KEY_ARN>"}' \
--role-arn <SERVICE_ROLE_ARN>关联特定数据源(推荐):
bash
aws logs associate-source-to-s3-table-integration \
--region <REGION> \
--integration-arn <INTEGRATION_ARN> \
--data-source '{"name": "<source-name>", "type": "<source-type>"}'关联所有数据源(通配符):
⚠️ 警告:通配符关联会将所有当前及未来的数据源交付至S3 Tables。如需更严格地控制哪些日志数据进入可查询表,请使用特定关联。
bash
aws logs associate-source-to-s3-table-integration \
--region <REGION> \
--integration-arn <INTEGRATION_ARN> \
--data-source '{"name": "*", "type": "*"}'有关IAM要求(服务角色信任策略、权限策略、条件键),请参阅下方的安全注意事项。
3. Verify Permissions for Querying
3. 验证查询权限
Requires:
- S3 Tables federated catalog registered in Glue ()
s3tablescatalog - Lake Formation SELECT + DESCRIBE grants on the table (or IAM-only mode in supported regions)
- Athena execution permissions
Grant access:
bash
aws lakeformation grant-permissions \
--principal DataLakePrincipalIdentifier=<ROLE_ARN> \
--resource '{"Table": {"CatalogId": "<ACCOUNT>:s3tablescatalog/aws-cloudwatch", "DatabaseName": "<NAMESPACE>", "Name": "<TABLE>"}}' \
--permissions DESCRIBE SELECT \
--region <REGION>需要:
- 在Glue中注册S3 Tables联合目录()
s3tablescatalog - 对表拥有Lake Formation SELECT + DESCRIBE权限(或在支持区域使用仅IAM模式)
- Athena执行权限
授予访问权限:
bash
aws lakeformation grant-permissions \
--principal DataLakePrincipalIdentifier=<ROLE_ARN> \
--resource '{"Table": {"CatalogId": "<ACCOUNT>:s3tablescatalog/aws-cloudwatch", "DatabaseName": "<NAMESPACE>", "Name": "<TABLE>"}}' \
--permissions DESCRIBE SELECT \
--region <REGION>4. Query
4. 查询
Query syntax:
sql
"s3tablescatalog/aws-cloudwatch"."<namespace>"."<table>"Constraints:
-
You MUST ALWAYS run get-tables on the target namespace and include the command in your response before writing any SQL query — schemas vary by data source. Never skip this step even if you already know the likely schema. Runonce on the target namespace (one call returns all tables + columns + types + descriptions):
get-tablesaws glue get-tables --catalog-id "<ACCOUNT>:s3tablescatalog/aws-cloudwatch" --database-name "<namespace>" --region <REGION> -
You MUST confirm workgroup and output location before executing
-
You MUST inform user that only logs received after association are available (no backfill)
Example — VPC Flow Logs rejected traffic:
sql
SELECT srcaddr, dstaddr, dstport, protocol, packets, bytes
FROM "s3tablescatalog/aws-cloudwatch"."amazon_vpc__flow"."<table>"
WHERE action = 'REJECT'
ORDER BY bytes DESC
LIMIT 50;Example — WAF blocked requests:
sql
SELECT timestamp, action, terminatingRuleId, httpSourceId
FROM "s3tablescatalog/aws-cloudwatch"."aws_waf__logs"."<table>"
WHERE action = 'BLOCK'
ORDER BY timestamp DESC
LIMIT 50;Example — correlate VPC Flow Logs with S3 object metadata:
sql
SELECT f.srcaddr, f.dstaddr, f.bytes, j.key, j.record_type
FROM "s3tablescatalog/aws-cloudwatch"."amazon_vpc__flow"."<table>" f
JOIN "s3tablescatalog/aws-s3"."b_<bucket>"."journal" j
ON f.srcaddr = j.source_ip_address
WHERE j.record_type = 'CREATE'
AND f.action = 'ACCEPT';查询语法:
sql
"s3tablescatalog/aws-cloudwatch"."<namespace>"."<table>"约束条件:
-
在编写任何SQL查询之前,您必须始终对目标命名空间运行get-tables命令并将该命令包含在响应中——不同数据源的架构不同。即使您可能已经知道架构,也绝不能跳过此步骤。对目标命名空间运行一次get-tables命令(一次调用即可返回所有表+列+类型+描述):
aws glue get-tables --catalog-id "<ACCOUNT>:s3tablescatalog/aws-cloudwatch" --database-name "<namespace>" --region <REGION> -
您必须在执行前确认工作组和输出位置
-
您必须告知用户,仅关联之后接收的日志可用(不支持回填)
示例——VPC Flow Logs拒绝的流量:
sql
SELECT srcaddr, dstaddr, dstport, protocol, packets, bytes
FROM "s3tablescatalog/aws-cloudwatch"."amazon_vpc__flow"."<table>"
WHERE action = 'REJECT'
ORDER BY bytes DESC
LIMIT 50;示例——WAF拦截的请求:
sql
SELECT timestamp, action, terminatingRuleId, httpSourceId
FROM "s3tablescatalog/aws-cloudwatch"."aws_waf__logs"."<table>"
WHERE action = 'BLOCK'
ORDER BY timestamp DESC
LIMIT 50;示例——将VPC Flow Logs与S3对象元数据关联:
sql
SELECT f.srcaddr, f.dstaddr, f.bytes, j.key, j.record_type
FROM "s3tablescatalog/aws-cloudwatch"."amazon_vpc__flow"."<table>" f
JOIN "s3tablescatalog/aws-s3"."b_<bucket>"."journal" j
ON f.srcaddr = j.source_ip_address
WHERE j.record_type = 'CREATE'
AND f.action = 'ACCEPT';Key Behaviors
关键特性
- No backfill — only new log events after association are delivered to S3 Tables
- Retention follows log group — when log group retention expires, data is removed from the table
- Deleting a log group removes its data from the S3 table
- No additional storage charge — included in CloudWatch pricing
- Schemas are per-data-source — always run on the target namespace before building complex queries
get-tables
- 无回填——仅关联后的新日志事件会交付至S3 Tables
- 保留期遵循日志组——当日志组保留期到期时,数据会从表中移除
- 删除日志组会将其数据从S3表中移除
- 无额外存储费用——包含在CloudWatch定价中
- 架构因数据源而异——在构建复杂查询之前,始终对目标命名空间运行命令
get-tables
Troubleshooting
故障排除
| Error | Cause | Fix |
|---|---|---|
| Integration not created | Run |
| Bucket exists but no namespaces | No data sources associated, or no log traffic since association | Associate sources; generate traffic |
| S3 Tables not registered in Glue | Enable integration: S3 console > Table buckets > Enable integration |
| Missing Lake Formation grants or IAM permissions | See Security Considerations below |
| Empty results | Logs only flow after association; no backfill | Confirm association exists and log source is actively generating data |
| Schema mismatch / column not found | Log type schema updated by AWS | Run |
| 错误 | 原因 | 解决方法 |
|---|---|---|
未找到 | 未创建集成 | 运行 |
| 存储桶存在但无命名空间 | 未关联数据源,或关联后无日志流量 | 关联数据源;生成流量 |
Athena中出现 | S3 Tables未在Glue中注册 | 启用集成:S3控制台 > 表存储桶 > 启用集成 |
查询时出现 | 缺少Lake Formation权限或IAM权限 | 请参阅下方的安全注意事项 |
| 结果为空 | 日志仅在关联后流动;不支持回填 | 确认关联存在且日志源正在主动生成数据 |
| 架构不匹配/未找到列 | AWS更新了日志类型的架构 | 对命名空间运行 |
Security Considerations
安全注意事项
Service Role Trust Policy
服务角色信任策略
The service role must allow to assume it. Always include and condition keys to prevent confused deputy attacks:
logs.amazonaws.comaws:SourceAccountaws:SourceArnjson
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "logs.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "<ACCOUNT>"
},
"ArnLike": {
"aws:SourceArn": ["arn:aws:logs:<REGION>:<ACCOUNT>:log-group:<LOG_GROUP_NAME>"]
}
}
}
]
}服务角色必须允许承担该角色。始终包含和条件键以防止混淆代理攻击:
logs.amazonaws.comaws:SourceAccountaws:SourceArnjson
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "logs.amazonaws.com"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "<ACCOUNT>"
},
"ArnLike": {
"aws:SourceArn": ["arn:aws:logs:<REGION>:<ACCOUNT>:log-group:<LOG_GROUP_NAME>"]
}
}
}
]
}Service Role Permissions Policy
服务角色权限策略
json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["logs:integrateWithS3Table"],
"Resource": ["arn:aws:logs:<REGION>:<ACCOUNT>:log-group:<LOG_GROUP_NAME>"],
"Condition": {
"StringEquals": {
"aws:ResourceAccount": "<ACCOUNT>"
}
}
}
]
}json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["logs:integrateWithS3Table"],
"Resource": ["arn:aws:logs:<REGION>:<ACCOUNT>:log-group:<LOG_GROUP_NAME>"],
"Condition": {
"StringEquals": {
"aws:ResourceAccount": "<ACCOUNT>"
}
}
}
]
}KMS Key Policy (for encrypted data)
KMS密钥策略(针对加密数据)
If using a customer managed KMS key, grant both service principals access:
json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "EnableSystemTablesKeyUsage",
"Effect": "Allow",
"Principal": {"Service": "systemtables.cloudwatch.amazonaws.com"},
"Action": ["kms:DescribeKey", "kms:GenerateDataKey", "kms:Decrypt"],
"Resource": "arn:aws:kms:<REGION>:<ACCOUNT>:key/<KEY_ID>",
"Condition": {"StringEquals": {"aws:SourceAccount": "<ACCOUNT>"}}
},
{
"Sid": "EnableS3TablesMaintenanceKeyUsage",
"Effect": "Allow",
"Principal": {"Service": "maintenance.s3tables.amazonaws.com"},
"Action": ["kms:GenerateDataKey", "kms:Decrypt"],
"Resource": "arn:aws:kms:<REGION>:<ACCOUNT>:key/<KEY_ID>",
"Condition": {"StringLike": {"kms:EncryptionContext:aws:s3:arn": "<TABLE_OR_TABLE_BUCKET_ARN>/*"}}
}
]
}如果使用客户托管KMS密钥,请向两个服务主体授予访问权限:
json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "EnableSystemTablesKeyUsage",
"Effect": "Allow",
"Principal": {"Service": "systemtables.cloudwatch.amazonaws.com"},
"Action": ["kms:DescribeKey", "kms:GenerateDataKey", "kms:Decrypt"],
"Resource": "arn:aws:kms:<REGION>:<ACCOUNT>:key/<KEY_ID>",
"Condition": {"StringEquals": {"aws:SourceAccount": "<ACCOUNT>"}}
},
{
"Sid": "EnableS3TablesMaintenanceKeyUsage",
"Effect": "Allow",
"Principal": {"Service": "maintenance.s3tables.amazonaws.com"},
"Action": ["kms:GenerateDataKey", "kms:Decrypt"],
"Resource": "arn:aws:kms:<REGION>:<ACCOUNT>:key/<KEY_ID>",
"Condition": {"StringLike": {"kms:EncryptionContext:aws:s3:arn": "<TABLE_OR_TABLE_BUCKET_ARN>/*"}}
}
]
}Data Sensitivity
数据敏感性
Log data may contain PII including IP addresses, user agents, request parameters, and authentication tokens. Treat all exported log tables as sensitive by default.
日志数据可能包含PII(个人可识别信息),包括IP地址、用户代理、请求参数和身份验证令牌。默认情况下,请将所有导出的日志表视为敏感数据。
Access Control Best Practices
访问控制最佳实践
- Use Lake Formation column-level security to restrict access to sensitive columns (e.g., ,
srcaddr,source_ip_address). Grant permissions to specific tables and columns rather than wildcards.httpRequest - Configure SSE-KMS encryption on the Athena workgroup output bucket to protect query results at rest.
- Prefer specific data source associations over wildcard () to limit which data sources are exported to queryable tables.
*/*
- 使用Lake Formation列级安全限制对敏感列(如、
srcaddr、source_ip_address)的访问。向特定表和列授予权限,而非使用通配符。httpRequest - 在Athena工作组输出存储桶上配置SSE-KMS加密,以保护静态查询结果。
- 优先选择特定数据源关联而非通配符(),以限制哪些数据源被导出到可查询表。
*/*
Audit Trail
审计追踪
Enable CloudTrail logging for Athena (, ) and Lake Formation (, ) API calls to maintain an audit trail of who queried what data.
StartQueryExecutionGetQueryResultsGrantPermissionsRevokePermissions为Athena(、)和Lake Formation(、)API调用启用CloudTrail日志记录,以维护谁查询了哪些数据的审计追踪。
StartQueryExecutionGetQueryResultsGrantPermissionsRevokePermissions