ai-monitoring

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Routing note: For ambiguous user intents, use the shared clarification templates in references/intent-clarification.md.

路由说明：若用户意图不明确，请使用references/intent-clarification.md中的通用澄清模板。

AI Monitoring

AI监控

Query AI Gateway request traces, costs, latency, errors, and token usage via the spans query API.

通过spans查询API查询AI Gateway的请求跟踪、成本、延迟、错误及令牌使用情况。

When to Use

适用场景

Investigate gateway traffic: recent requests, cost breakdowns, error rates, model usage, per-user activity, MCP tool calls, or latency analysis.

调查网关流量：近期请求、成本细分、错误率、模型使用情况、单用户活动、MCP工具调用或延迟分析。

When NOT to Use

不适用场景

User wants to instrument their own application with tracing -> prefer
```
tracing
```
skill (this skill is for querying existing gateway traces, not adding instrumentation)
User wants to configure gateway models, routing, or rate limits -> prefer
```
ai-gateway
```
skill
User wants to view application container logs -> prefer
```
logs
```
skill
User wants to check platform connectivity -> prefer
```
status
```
skill

</objective> <instructions>

用户想为自己的应用添加跟踪工具 -> 优先使用
```
tracing
```
技能（本技能用于查询现有网关跟踪数据，而非添加埋点）
用户想配置网关模型、路由或速率限制 -> 优先使用
```
ai-gateway
```
技能
用户想查看应用容器日志 -> 优先使用
```
logs
```
技能
用户想检查平台连通性 -> 优先使用
```
status
```
技能

</objective> <instructions>

Prerequisites

前置条件

Run the

status

skill first to confirm

TFY_BASE_URL

and

TFY_API_KEY

are set and valid.

When using direct API, set

TFY_API_SH

to the full path of this skill's

scripts/tfy-api.sh

. See

references/tfy-api-setup.md

for paths per agent.

首先运行

status

技能，确认

TFY_BASE_URL

和

TFY_API_KEY

已设置且有效。

使用直接API时，需将

TFY_API_SH

设置为该技能

scripts/tfy-api.sh

的完整路径。各Agent对应的路径请参考

references/tfy-api-setup.md

。

Required Parameter

必填参数

Every query requires one of these two parameters. Ask the user which one to use:

Parameter	Description
`tracingProjectFqn`	Fully qualified name of the tracing project, e.g. `tenant:tracing-project:name`
`dataRoutingDestination`	Data routing destination name, e.g. `default`

If the user does not know which to use, suggest

"dataRoutingDestination": "default"

as a starting point.

每次查询需要以下两个参数中的一个。请询问用户使用哪一个：

参数	描述
`tracingProjectFqn`	跟踪项目的完全限定名称，例如 `tenant:tracing-project:name`
`dataRoutingDestination`	数据路由目标名称，例如 `default`

如果用户不知道使用哪个，建议从

"dataRoutingDestination": "default"

开始尝试。

Query Spans API

查询Spans API

Endpoint:

POST /api/svc/v1/spans/query

接口地址：

POST /api/svc/v1/spans/query

Via Direct API

通过直接API调用

bash

undefined

bash

undefined

Set the path to tfy-api.sh for your agent (example for Claude Code):

为你的Agent设置tfy-api.sh的路径（以Claude Code为例）：

TFY_API_SH=~/.claude/skills/truefoundry-ai-monitoring/scripts/tfy-api.sh

Basic query: recent spans in the last 24 hours

基础查询：过去24小时内的近期跟踪数据

$TFY_API_SH POST '/api/svc/v1/spans/query' '{ "startTime": "2026-03-26T00:00:00.000Z", "endTime": "2026-03-27T00:00:00.000Z", "dataRoutingDestination": "default", "limit": 50, "sortDirection": "desc" }'

undefined

undefined

Common Use Cases

常见使用场景

1. Show Recent Requests

1. 查看近期请求

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "limit": 20,
  "sortDirection": "desc"
}'

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "limit": 20,
  "sortDirection": "desc"
}'

2. Cost Analysis (LLM Spans)

2. 成本分析（LLM跟踪数据）

Filter for LLM spans and extract cost attributes:

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "filters": [
    {"spanAttributeKey": "tfy.span_type", "operator": "eq", "value": "LLM"}
  ],
  "limit": 200,
  "sortDirection": "desc"
}'

Cost fields in

spanAttributes

```
gen_ai.usage.cost
```
or
```
tfy.request_cost
```
-- cost of the request
```
gen_ai.usage.input_tokens
```
-- input token count
```
gen_ai.usage.output_tokens
```
-- output token count

筛选LLM跟踪数据并提取成本属性：

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "filters": [
    {"spanAttributeKey": "tfy.span_type", "operator": "eq", "value": "LLM"}
  ],
  "limit": 200,
  "sortDirection": "desc"
}'

spanAttributes

中的成本字段：

```
gen_ai.usage.cost
```
或
```
tfy.request_cost
```
-- 请求成本
```
gen_ai.usage.input_tokens
```
-- 输入令牌数量
```
gen_ai.usage.output_tokens
```
-- 输出令牌数量

3. Show Errors

3. 查看错误

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "filters": [
    {"spanFieldName": "statusCode", "operator": "eq", "value": "ERROR"}
  ],
  "limit": 50,
  "sortDirection": "desc"
}'

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "filters": [
    {"spanFieldName": "statusCode", "operator": "eq", "value": "ERROR"}
  ],
  "limit": 50,
  "sortDirection": "desc"
}'

4. Model Usage Breakdown

4. 模型使用情况细分

Query all LLM spans and extract model info from span attributes to see which models are being used:

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "filters": [
    {"spanAttributeKey": "tfy.span_type", "operator": "eq", "value": "LLM"}
  ],
  "limit": 200,
  "sortDirection": "desc"
}'

Parse

spanAttributes

in the response for model name fields.

查询所有LLM跟踪数据，从跟踪属性中提取模型信息，查看正在使用的模型：

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "filters": [
    {"spanAttributeKey": "tfy.span_type", "operator": "eq", "value": "LLM"}
  ],
  "limit": 200,
  "sortDirection": "desc"
}'

解析响应中的

spanAttributes

获取模型名字段。

5. Requests by a Specific User

5. 特定用户的请求

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "createdBySubjectSlugs": ["user@example.com"],
  "limit": 50,
  "sortDirection": "desc"
}'

You can also filter by subject type:

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "createdBySubjectTypes": ["virtualaccount"],
  "limit": 50,
  "sortDirection": "desc"
}'

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "createdBySubjectSlugs": ["user@example.com"],
  "limit": 50,
  "sortDirection": "desc"
}'

你也可以按主体类型筛选：

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "createdBySubjectTypes": ["virtualaccount"],
  "limit": 50,
  "sortDirection": "desc"
}'

6. MCP Tool Calls

6. MCP工具调用

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "filters": [
    {"spanAttributeKey": "tfy.span_type", "operator": "eq", "value": "MCP"}
  ],
  "limit": 50,
  "sortDirection": "desc"
}'

For MCP Gateway spans use

"value": "MCPGateway"

instead.

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "filters": [
    {"spanAttributeKey": "tfy.span_type", "operator": "eq", "value": "MCP"}
  ],
  "limit": 50,
  "sortDirection": "desc"
}'

对于MCP Gateway跟踪数据，请使用

"value": "MCPGateway"

替代。

7. Filter by Application Name

7. 按应用名称筛选

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "applicationNames": ["tfy-llm-gateway"],
  "limit": 50,
  "sortDirection": "desc"
}'

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "applicationNames": ["tfy-llm-gateway"],
  "limit": 50,
  "sortDirection": "desc"
}'

8. Filter by Span Name (endpoint pattern)

8. 按跟踪名称筛选（端点模式）

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "filters": [
    {"spanFieldName": "spanName", "operator": "contains", "value": "completions"}
  ],
  "limit": 50,
  "sortDirection": "desc"
}'

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "filters": [
    {"spanFieldName": "spanName", "operator": "contains", "value": "completions"}
  ],
  "limit": 50,
  "sortDirection": "desc"
}'

9. Filter by Gateway Request Metadata

9. 按网关请求元数据筛选

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "filters": [
    {"gatewayRequestMetadataKey": "tfy_gateway_region", "operator": "eq", "value": "US"}
  ],
  "limit": 50,
  "sortDirection": "desc"
}'

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "filters": [
    {"gatewayRequestMetadataKey": "tfy_gateway_region", "operator": "eq", "value": "US"}
  ],
  "limit": 50,
  "sortDirection": "desc"
}'

Request Body Reference

请求体参考

Field	Type	Required	Description
`startTime`	string (ISO 8601)	Yes	Start of time range
`endTime`	string (ISO 8601)	No	End of time range (defaults to now)
`tracingProjectFqn`	string	One of this or `dataRoutingDestination`	Tracing project FQN
`dataRoutingDestination`	string	One of this or `tracingProjectFqn`	Data routing destination
`traceIds`	string[]	No	Filter by trace IDs
`spanIds`	string[]	No	Filter by span IDs
`parentSpanIds`	string[]	No	Filter by parent span IDs
`createdBySubjectTypes`	string[]	No	Filter by subject type ( `user` , `virtualaccount` )
`createdBySubjectSlugs`	string[]	No	Filter by subject slug (e.g. email)
`applicationNames`	string[]	No	Filter by application name
`limit`	integer	No	Max results (default 200)
`sortDirection`	string	No	`asc` or `desc`
`pageToken`	string	No	Pagination token from previous response
`filters`	array	No	Array of filter objects (see Filter Types)
`includeFeedbacks`	boolean	No	Include feedback data

字段	类型	是否必填	描述
`startTime`	string (ISO 8601)	是	时间范围的起始时间
`endTime`	string (ISO 8601)	否	时间范围的结束时间（默认当前时间）
`tracingProjectFqn`	string	与 `dataRoutingDestination` 二选一	跟踪项目的完全限定名称
`dataRoutingDestination`	string	与 `tracingProjectFqn` 二选一	数据路由目标
`traceIds`	string[]	否	按跟踪ID筛选
`spanIds`	string[]	否	按跟踪数据ID筛选
`parentSpanIds`	string[]	否	按父跟踪数据ID筛选
`createdBySubjectTypes`	string[]	否	按主体类型筛选（ `user` 、 `virtualaccount` ）
`createdBySubjectSlugs`	string[]	否	按主体标识筛选（如邮箱）
`applicationNames`	string[]	否	按应用名称筛选
`limit`	integer	否	最大结果数（默认200）
`sortDirection`	string	否	`asc` 或 `desc`
`pageToken`	string	否	上一次响应中的分页令牌
`filters`	array	否	筛选对象数组（见筛选类型）
`includeFeedbacks`	boolean	否	是否包含反馈数据

Filter Types

筛选类型

SpanFieldFilter

json

{"spanFieldName": "<field>", "operator": "<op>", "value": "<val>"}

Fields:

spanName

serviceName

spanKind

statusCode

, etc.

json

{"spanFieldName": "<field>", "operator": "<op>", "value": "<val>"}

字段：

spanName

、

serviceName

、

spanKind

、

statusCode

等。

SpanAttributeFilter

json

{"spanAttributeKey": "<key>", "operator": "<op>", "value": "<val>"}

Any key from the

spanAttributes

dict (e.g.

tfy.span_type

gen_ai.usage.cost

json

{"spanAttributeKey": "<key>", "operator": "<op>", "value": "<val>"}

spanAttributes

字典中的任意键（例如

tfy.span_type

、

gen_ai.usage.cost

）。

GatewayRequestMetadataFilter

json

{"gatewayRequestMetadataKey": "<key>", "operator": "<op>", "value": "<val>"}

Custom metadata keys set via

X-TFY-LOGGING-CONFIG

headers.

json

{"gatewayRequestMetadataKey": "<key>", "operator": "<op>", "value": "<val>"}

通过

X-TFY-LOGGING-CONFIG

头设置的自定义元数据键。

Filter Operators

筛选操作符

eq

neq

contains

not_contains

starts_with

ends_with

eq

、

neq

、

contains

、

not_contains

、

starts_with

、

ends_with

Response Structure

响应结构

json

{
  "data": [
    {
      "spanId": "...",
      "traceId": "...",
      "parentSpanId": "...",
      "serviceName": "tfy-llm-gateway",
      "spanName": "POST https://api.openai.com/v1/chat/completions",
      "spanKind": "Client",
      "scopeName": "...",
      "scopeVersion": "...",
      "timestamp": "2026-03-26T14:30:00.000Z",
      "durationNs": 1234567890,
      "statusCode": "OK",
      "statusMessage": "",
      "spanAttributes": {
        "gen_ai.usage.input_tokens": 150,
        "gen_ai.usage.output_tokens": 80,
        "gen_ai.usage.cost": 0.0023,
        "tfy.request_cost": 0.0023,
        "tfy.span_type": "LLM"
      },
      "events": [],
      "createdBySubject": {
        "subjectId": "...",
        "subjectSlug": "user@example.com",
        "subjectType": "user",
        "tenantName": "my-tenant"
      },
      "feedbacks": []
    }
  ],
  "pagination": {
    "nextPageToken": "..."
  }
}

json

{
  "data": [
    {
      "spanId": "...",
      "traceId": "...",
      "parentSpanId": "...",
      "serviceName": "tfy-llm-gateway",
      "spanName": "POST https://api.openai.com/v1/chat/completions",
      "spanKind": "Client",
      "scopeName": "...",
      "scopeVersion": "...",
      "timestamp": "2026-03-26T14:30:00.000Z",
      "durationNs": 1234567890,
      "statusCode": "OK",
      "statusMessage": "",
      "spanAttributes": {
        "gen_ai.usage.input_tokens": 150,
        "gen_ai.usage.output_tokens": 80,
        "gen_ai.usage.cost": 0.0023,
        "tfy.request_cost": 0.0023,
        "tfy.span_type": "LLM"
      },
      "events": [],
      "createdBySubject": {
        "subjectId": "...",
        "subjectSlug": "user@example.com",
        "subjectType": "user",
        "tenantName": "my-tenant"
      },
      "feedbacks": []
    }
  ],
  "pagination": {
    "nextPageToken": "..."
  }
}

Pagination

分页

When the response includes

pagination.nextPageToken

, pass it as

pageToken

in the next request to fetch the next page:

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "limit": 200,
  "pageToken": "TOKEN_FROM_PREVIOUS_RESPONSE"
}'

Continue until

nextPageToken

is null or absent.

当响应包含

pagination.nextPageToken

时，将其作为

pageToken

传入下一次请求以获取下一页数据：

bash

$TFY_API_SH POST '/api/svc/v1/spans/query' '{
  "startTime": "2026-03-26T00:00:00.000Z",
  "dataRoutingDestination": "default",
  "limit": 200,
  "pageToken": "TOKEN_FROM_PREVIOUS_RESPONSE"
}'

持续请求直到

nextPageToken

为null或不存在。

Presenting Results

结果展示

Format results as tables for readability:

Recent Gateway Requests (last 24h):
| Time                | Model          | Status | Tokens (in/out) | Cost     | Latency   | User              |
|---------------------|----------------|--------|-----------------|----------|-----------|-------------------|
| 2026-03-26 14:30:00 | openai/gpt-4o  | OK     | 150 / 80        | $0.0023  | 1.23s     | user@example.com  |
| 2026-03-26 14:29:55 | anthropic/...  | OK     | 200 / 120       | $0.0045  | 2.10s     | bot@svc           |
| 2026-03-26 14:29:30 | openai/gpt-4o  | ERROR  | 100 / 0         | $0.0000  | 0.45s     | user@example.com  |

For cost summaries, aggregate across spans:

Cost Summary (last 24h):
| Model              | Requests | Total Cost | Avg Cost/Req | Total Tokens |
|--------------------|----------|------------|--------------|--------------|
| openai/gpt-4o      | 142      | $3.21      | $0.023       | 45,200       |
| anthropic/claude    | 58       | $1.87      | $0.032       | 22,100       |
| Total               | 200      | $5.08      | $0.025       | 67,300       |

Convert

durationNs

(nanoseconds) to human-readable format: divide by 1,000,000,000 for seconds.

</instructions>

<success_criteria>

将结果格式化为表格以提升可读性：

近期网关请求（过去24小时）：
| 时间                | 模型          | 状态 | 令牌数（入/出） | 成本     | 延迟   | 用户              |
|---------------------|----------------|--------|-----------------|----------|-----------|-------------------|
| 2026-03-26 14:30:00 | openai/gpt-4o  | OK     | 150 / 80        | $0.0023  | 1.23s     | user@example.com  |
| 2026-03-26 14:29:55 | anthropic/...  | OK     | 200 / 120       | $0.0045  | 2.10s     | bot@svc           |
| 2026-03-26 14:29:30 | openai/gpt-4o  | ERROR  | 100 / 0         | $0.0000  | 0.45s     | user@example.com  |

对于成本汇总，需对所有跟踪数据进行聚合：

成本汇总（过去24小时）：
| 模型              | 请求数 | 总成本 | 平均每次请求成本 | 总令牌数 |
|--------------------|----------|------------|--------------|--------------|
| openai/gpt-4o      | 142      | $3.21      | $0.023       | 45,200       |
| anthropic/claude    | 58       | $1.87      | $0.032       | 22,100       |
| 总计               | 200      | $5.08      | $0.025       | 67,300       |

将

durationNs

（纳秒）转换为人类可读格式：除以1,000,000,000得到秒数。

</instructions>

<success_criteria>

Success Criteria

成功标准

The user can see recent AI Gateway request traces with timestamps, models, status, and costs
Cost and token usage are summarized clearly with per-model breakdowns when requested
Errors are identified with status codes and messages for debugging
Results are presented as formatted tables, not raw JSON
Pagination is handled correctly for large result sets
The agent asked for
```
dataRoutingDestination
```
or
```
tracingProjectFqn
```
before querying

</success_criteria>

用户可查看包含时间戳、模型、状态和成本的近期AI Gateway请求跟踪数据
成本和令牌使用情况按模型细分并清晰汇总（若用户要求）
错误信息包含状态码和消息，便于调试
结果以格式化表格展示，而非原始JSON
针对大型结果集正确处理分页
Agent在查询前询问用户
```
dataRoutingDestination
```
或
```
tracingProjectFqn
```
参数

</success_criteria>

Composability

组合使用

Preflight check: Use
```
status
```
skill to verify credentials before querying
Gateway configuration: Use
```
ai-gateway
```
skill to configure models, routing, rate limits
Instrument your app: Use
```
tracing
```
skill to add tracing to your own applications (different from monitoring existing gateway traces)
View container logs: Use
```
logs
```
skill for application-level logs (not gateway request traces)
Manage access tokens: Use
```
access-tokens
```
skill to create/manage PAT or VAT used for gateway auth

</references> <troubleshooting>

预检检查：查询前使用
```
status
```
技能验证凭证有效性
网关配置：使用
```
ai-gateway
```
技能配置模型、路由和速率限制
应用埋点：使用
```
tracing
```
技能为自有应用添加跟踪（与监控现有网关跟踪数据不同）
查看容器日志：使用
```
logs
```
技能查看应用级日志（非网关请求跟踪数据）
管理访问令牌：使用
```
access-tokens
```
技能创建/管理用于网关认证的PAT或VAT

</references> <troubleshooting>

Error Handling

错误处理

400 Bad Request

Missing required parameter. Ensure you provide either:
- "tracingProjectFqn": "tenant:tracing-project:name"
- "dataRoutingDestination": "default"
And a valid "startTime" in ISO 8601 format.

缺少必填参数。请确保提供以下参数之一：
- "tracingProjectFqn": "tenant:tracing-project:name"
- "dataRoutingDestination": "default"
并提供ISO 8601格式的有效"startTime"。

401 Unauthorized

Authentication failed. Run the status skill to verify your TFY_API_KEY is valid.

认证失败。运行status技能验证你的TFY_API_KEY是否有效。

No Data Returned

无数据返回

Empty results. Check:
- Time range is correct (startTime/endTime)
- The dataRoutingDestination or tracingProjectFqn exists
- Filters are not too restrictive (try removing filters first)
- Gateway has actually received requests in this time period

结果为空。请检查：
- 时间范围是否正确（startTime/endTime）
- dataRoutingDestination或tracingProjectFqn是否存在
- 筛选条件是否过于严格（先尝试移除筛选条件）
- 网关在此时间段内是否确实收到了请求

Pagination Token Expired

分页令牌过期

If a pageToken returns an error, restart the query from the beginning
with a fresh request (no pageToken).

</troubleshooting> </output>

若pageToken返回错误，请从新请求开始重新查询（不携带pageToken）。

</troubleshooting>