inngest-agent-evals

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Inngest Agent Evals

Use this skill when the user wants to evaluate AI agents or AI workflows in production, add scoring, compare prompts/models/tools, group related runs, or debug why an agent outcome was good or bad.

Agent Evals is not a separate package. It is the production evaluation workflow built from Inngest functions, durable steps, scores, deferred scorers, sessions, traces, experiments, and Insights.

当用户需要在生产环境中评估AI代理或AI工作流、添加评分机制、对比提示词/模型/工具、分组相关运行实例，或调试代理结果优劣的原因时，可使用本技能。

Agent Evals并非独立包，它是基于Inngest函数、持久化步骤、评分、延迟评分器、会话、追踪、实验和Insights构建的生产环境评估工作流。

Canonical References

标准参考文档

Check the public docs first when exact API details matter:

Agent Evals overview: https://www.inngest.com/docs/learn/agent-evals
Scoring guide: https://www.inngest.com/docs/features/inngest-functions/steps-workflows/scoring
Deferred scoring guide: https://www.inngest.com/docs/features/inngest-functions/steps-workflows/deferred-scoring
Step experiments guide: https://www.inngest.com/docs/features/inngest-functions/steps-workflows/step-experiments
Sessions: https://www.inngest.com/docs/features/events-triggers/sessions
Traces: https://www.inngest.com/docs/platform/monitor/traces
Insights: https://www.inngest.com/docs/platform/monitor/insights
Scoring reference: https://www.inngest.com/docs/reference/typescript/v4/functions/scoring
```
group.experiment()
```
reference: https://www.inngest.com/docs/reference/typescript/v4/functions/group-experiment
Launch context: https://www.inngest.com/blog/introducing-agent-evals

当需要精确API细节时，请先查阅公开文档：

Agent Evals概述：https://www.inngest.com/docs/learn/agent-evals
评分指南：https://www.inngest.com/docs/features/inngest-functions/steps-workflows/scoring
延迟评分指南：https://www.inngest.com/docs/features/inngest-functions/steps-workflows/deferred-scoring
步骤实验指南：https://www.inngest.com/docs/features/inngest-functions/steps-workflows/step-experiments
会话：https://www.inngest.com/docs/features/events-triggers/sessions
追踪：https://www.inngest.com/docs/platform/monitor/traces
Insights：https://www.inngest.com/docs/platform/monitor/insights
评分参考：https://www.inngest.com/docs/reference/typescript/v4/functions/scoring
```
group.experiment()
```
参考：https://www.inngest.com/docs/reference/typescript/v4/functions/group-experiment
发布背景：https://www.inngest.com/blog/introducing-agent-evals

Decision Flow

决策流程

Start from the outcome, not the mechanism:

Identify the product or quality signal: helpful click, ticket resolution, conversion, guardrail pass, retrieval quality, model confidence, latency, cost, human review, or LLM-as-judge result.
Decide when the signal appears:
- During the run: use direct scoring with
```
step.score()
```
  or
```
inngest.score()
```
  .
- After the run: use deferred scoring with
```
createScorer()
```
  and
```
defer()
```
  , or explicit
```
inngest.score({ runId })
```
  .
Decide how humans will inspect related work:
- Use
```
meta.sessions
```
  for repeated high-cardinality identifiers such as
```
conversation_id
```
  ,
```
ticket_id
```
  ,
```
agent_run_id
```
  , or
```
import_id
```
  .
- Use Insights for ad hoc historical analysis, low-cardinality filters, SQL aggregates, and broad investigation.
Decide whether to compare variants:
- Use
```
group.experiment()
```
  for prompt, model, provider, tool, workflow, or rollout comparisons.
- Pass
```
experimentRef
```
  to deferred or later scores when the score should be credited to the selected variant.
Preserve traceability:
- Keep model calls, tool calls, waits, database writes, and scoring work in durable steps so traces explain why the score happened.

从结果出发，而非机制：

确定产品或质量信号：有效点击、工单解决、转化、合规检查通过、检索质量、模型置信度、延迟、成本、人工审核或LLM作为评判者的结果。
判断信号出现的时机：
- 运行过程中：使用
```
step.score()
```
  或
```
inngest.score()
```
  进行直接评分。
- 运行结束后：使用
```
createScorer()
```
  和
```
defer()
```
  进行延迟评分，或显式调用
```
inngest.score({ runId })
```
  。
确定人工检查相关工作的方式：
- 对于重复出现的高基数标识符（如
```
conversation_id
```
  、
```
ticket_id
```
  、
```
agent_run_id
```
  或
```
import_id
```
  ），使用
```
meta.sessions
```
  。
- 对于临时历史分析、低基数过滤、SQL聚合和广泛调查，使用Insights。
判断是否需要对比变体：
- 对比提示词、模型、提供商、工具、工作流或版本发布时，使用
```
group.experiment()
```
  。
- 当评分需归属于选定变体时，将
```
experimentRef
```
  传递给延迟评分或后续评分。
保留可追溯性：
- 将模型调用、工具调用、等待、数据库写入和评分工作放在持久化步骤中，以便追踪记录能解释评分的由来。

Setup Requirements

配置要求

Use TypeScript SDK v4 and install the latest SDK:
```
npm install inngest@latest
```
.
Scoring and deferred scoring are beta APIs; verify current imports against the docs before shipping user-facing examples.

step.score()

requires

scoreMiddleware()

from

inngest/experimental

on the Inngest client.

```
createScorer()
```
is imported from
```
inngest/experimental
```
and the scorer must be registered in the serve handler with other functions.
Sessions sent through the TypeScript SDK require v4.7.0 or later.
Step experiments require v4.8.0 or later for
```
group.experiment()
```
and the
```
experiment
```
helper from
```
inngest
```
.

Client shape:

typescript

import { Inngest } from "inngest";
import { scoreMiddleware } from "inngest/experimental";

export const inngest = new Inngest({
  id: "support-agent",
  middleware: [scoreMiddleware()],
});

使用TypeScript SDK v4并安装最新版本：
```
npm install inngest@latest
```
。
评分和延迟评分属于beta API；在发布面向用户的示例前，请对照文档验证当前导入内容。

step.score()

需要在Inngest客户端上使用来自

inngest/experimental

的

scoreMiddleware()

。

```
createScorer()
```
从
```
inngest/experimental
```
导入，且评分器必须与其他函数一起在服务处理程序中注册。
通过TypeScript SDK发送会话需要v4.7.0或更高版本。
步骤实验需要v4.8.0或更高版本以支持
```
group.experiment()
```
和来自
```
inngest
```
的
```
experiment
```
工具。

客户端示例：

typescript

import { Inngest } from "inngest";
import { scoreMiddleware } from "inngest/experimental";

export const inngest = new Inngest({
  id: "support-agent",
  middleware: [scoreMiddleware()],
});

Sessions

会话

Add sessions when events belong to a user flow, conversation, ticket, import, or agent task that someone will inspect repeatedly.

typescript

await inngest.send({
  name: "support/ticket.created",
  data: {
    ticketId: "tk_123",
    message: "I can't sign in.",
  },
  meta: {
    sessions: {
      ticket_id: "tk_123",
    },
  },
});

Rules:

Use stable, non-secret, high-cardinality IDs.
Keep the key generic, such as
```
conversation_id
```
; put the actual ID in the value.
Pass sessions explicitly through
```
step.invoke()
```
and
```
step.sendEvent()
```
when downstream runs should join the same session.
Do not use sessions for labels like
```
environment: prod
```
; use Insights for that style of filtering.

当事件属于用户流程、对话、工单、导入任务或需要反复检查的代理任务时，添加会话。

typescript

await inngest.send({
  name: "support/ticket.created",
  data: {
    ticketId: "tk_123",
    message: "I can't sign in.",
  },
  meta: {
    sessions: {
      ticket_id: "tk_123",
    },
  },
});

规则：

使用稳定、非保密的高基数ID。
键名保持通用，如
```
conversation_id
```
；实际ID放在值中。
当下游运行实例需要加入同一会话时，通过
```
step.invoke()
```
和
```
step.sendEvent()
```
显式传递会话。
不要将会话用于
```
environment: prod
```
这类标签；此类过滤使用Insights。

Direct Scoring

直接评分

Use direct scoring when the score is known during the function run.

Good direct scores:

guardrail pass/fail
JSON validity
retrieval confidence
tool success
model confidence
inline LLM-as-judge result

typescript

export const answerTicket = inngest.createFunction(
  {
    id: "answer-ticket",
    triggers: [{ event: "support/ticket.created" }],
  },
  async ({ event, step }) => {
    const answer = await step.run("generate-answer", () =>
      generateAnswer(event.data.ticketId)
    );

    const passed = await step.run("check-answer", () => validateAnswer(answer));

    await step.score("score-answer-quality", {
      name: "answer-quality",
      value: passed,
    });

    return { answer, passed };
  }
);

Use stable score names. Changing a score name creates a separate metric. Score values must be finite numbers or booleans.

当评分在函数运行过程中即可确定时，使用直接评分。

适合直接评分的场景：

合规检查通过/失败
JSON有效性
检索置信度
工具调用成功
模型置信度
内置LLM作为评判者的结果

typescript

export const answerTicket = inngest.createFunction(
  {
    id: "answer-ticket",
    triggers: [{ event: "support/ticket.created" }],
  },
  async ({ event, step }) => {
    const answer = await step.run("generate-answer", () =>
      generateAnswer(event.data.ticketId)
    );

    const passed = await step.run("check-answer", () => validateAnswer(answer));

    await step.score("score-answer-quality", {
      name: "answer-quality",
      value: passed,
    });

    return { answer, passed };
  }
);

使用稳定的评分名称。更改评分名称会创建独立的指标。评分值必须是有限数字或布尔值。

Deferred Scoring

延迟评分

Use deferred scoring when the useful signal arrives after the parent workflow finishes, such as user feedback, conversion, ticket reopen, retention, or a slow LLM-as-judge run.

typescript

import { createScorer } from "inngest/experimental";
import { z } from "zod";

export const feedbackScorer = createScorer(
  inngest,
  {
    id: "support-feedback-scorer",
    schema: z.object({ ticketId: z.string() }),
  },
  async ({ event, step }) => {
    const feedback = await step.waitForEvent("wait-for-feedback", {
      event: "support/feedback.received",
      timeout: "7d",
      if: `async.data.ticketId == '${event.data.ticketId}'`,
    });

    return {
      name: "user-feedback",
      value: feedback?.data.helpful ? 1 : 0,
    };
  }
);

Trigger it from the producing function:

typescript

async ({ event, step, defer }) => {
  const answer = await step.run("generate-answer", () =>
    generateAnswer(event.data.ticketId)
  );

  defer("score-feedback", {
    function: feedbackScorer,
    data: { ticketId: event.data.ticketId },
  });

  return { answer };
}

Guardrails:

Register scorers with
```
serve({ functions: [...] })
```
.
```
defer()
```
is fire-and-forget; do not
```
await
```
it.
The parent run is attributed automatically for deferred scorers.
Return a default score or
```
null
```
when a signal times out, based on the product semantics.

当有用信号在父工作流结束后才出现时（如用户反馈、转化、工单重开、留存率或耗时较长的LLM评判运行），使用延迟评分。

typescript

import { createScorer } from "inngest/experimental";
import { z } from "zod";

export const feedbackScorer = createScorer(
  inngest,
  {
    id: "support-feedback-scorer",
    schema: z.object({ ticketId: z.string() }),
  },
  async ({ event, step }) => {
    const feedback = await step.waitForEvent("wait-for-feedback", {
      event: "support/feedback.received",
      timeout: "7d",
      if: `async.data.ticketId == '${event.data.ticketId}'`,
    });

    return {
      name: "user-feedback",
      value: feedback?.data.helpful ? 1 : 0,
    };
  }
);

从生产函数中触发：

typescript

async ({ event, step, defer }) => {
  const answer = await step.run("generate-answer", () =>
    generateAnswer(event.data.ticketId)
  );

  defer("score-feedback", {
    function: feedbackScorer,
    data: { ticketId: event.data.ticketId },
  });

  return { answer };
}

注意事项：

通过
```
serve({ functions: [...] })
```
注册评分器。
```
defer()
```
是“即发即忘”模式，不要使用
```
await
```
等待它。
延迟评分器会自动关联父运行实例。
根据产品语义，当信号超时返回默认评分或
```
null
```
。

Experiments

实验

Use

group.experiment()

when comparing prompts, models, providers, tools, workflow rewrites, or operational settings against real traffic.

typescript

import { experiment } from "inngest";

const { result, variant, experimentRef } = await group.experiment(
  "answer-style",
  {
    variants: {
      concise: () => step.run("answer-concise", () => answerConcise(event.data)),
      detailed: () =>
        step.run("answer-detailed", () => answerDetailed(event.data)),
    },
    select: experiment.bucket(event.data.accountId, {
      weights: { concise: 50, detailed: 50 },
    }),
  }
);

Selection strategy:

```
experiment.weighted()
```
for run-level traffic splits.
```
experiment.bucket(stableId, { weights })
```
when a user, account, or tenant should usually keep the same experience.
```
experiment.custom()
```
when assignment comes from a database, flag service, or rollout table.
```
experiment.fixed()
```
to force one variant during testing or after choosing a winner.

Rules:

Each variant callback must call at least one
```
step.*
```
tool.
Keep experiment IDs and variant names stable because they appear in traces.
The selected variant is memoized for retries and replays.
Persist
```
experimentRef
```
and the parent run ID if a later, separate process will score the selected variant.

When deferred scoring a variant, pass the ref:

typescript

defer("score-answer-feedback", {
  function: feedbackScorer,
  data: { ticketId: event.data.ticketId },
  experiment: experimentRef,
});

When scoring from a later run explicitly:

typescript

await inngest.score.experiment({
  name: "clickthrough",
  value: 1,
  experiment: experimentRef,
  runId: originalRunId,
});

当对比提示词、模型、提供商、工具、工作流重写或运营设置与真实流量时，使用

group.experiment()

。

typescript

import { experiment } from "inngest";

const { result, variant, experimentRef } = await group.experiment(
  "answer-style",
  {
    variants: {
      concise: () => step.run("answer-concise", () => answerConcise(event.data)),
      detailed: () =>
        step.run("answer-detailed", () => answerDetailed(event.data)),
    },
    select: experiment.bucket(event.data.accountId, {
      weights: { concise: 50, detailed: 50 },
    }),
  }
);

选择策略：

```
experiment.weighted()
```
用于运行级别的流量拆分。
当用户、账户或租户应保持一致体验时，使用
```
experiment.bucket(stableId, { weights })
```
。
当分配来自数据库、标记服务或发布表时，使用
```
experiment.custom()
```
。
测试期间或选定最优方案后，使用
```
experiment.fixed()
```
强制使用一个变体。

规则：

每个变体回调必须至少调用一个
```
step.*
```
工具。
保持实验ID和变体名称稳定，因为它们会出现在追踪记录中。
选定的变体在重试和重放时会被记忆。
如果后续独立进程需要为选定变体评分，请持久化
```
experimentRef
```
和父运行ID。

为变体进行延迟评分时，传递引用：

typescript

defer("score-answer-feedback", {
  function: feedbackScorer,
  data: { ticketId: event.data.ticketId },
  experiment: experimentRef,
});

从后续运行实例中显式评分时：

typescript

await inngest.score.experiment({
  name: "clickthrough",
  value: 1,
  experiment: experimentRef,
  runId: originalRunId,
});

Brownfield Migration

遗留系统迁移

When adding Agent Evals to an existing app:

Search for existing agent/workflow outputs and where users act on them.
Find durable identifiers already in the domain: conversation ID, ticket ID, account ID, import ID, run ID, or recommendation ID.
Add
```
meta.sessions
```
at event producers before building dashboards around correlation.
Add one direct score where the outcome is already known.
Add a deferred scorer only after the signal event exists or can be emitted cleanly.
Add
```
group.experiment()
```
only around one isolated decision at a time.
Keep old prompt/model/tool behavior stable until scoring proves the new path.

Use

inngest-brownfield-audit

first when the repo has many candidate workflows. Use

inngest-agents

with this skill when the work is an AgentKit or durable agent workflow.

在现有应用中添加Agent Evals时：

搜索现有的代理/工作流输出以及用户与之交互的位置。
找到领域中已有的持久化标识符：对话ID、工单ID、账户ID、导入ID、运行ID或推荐ID。
在构建基于关联的仪表板之前，先在事件生产者中添加
```
meta.sessions
```
。
在已知结果的位置添加一个直接评分。
仅在信号事件已存在或可清晰触发时，添加延迟评分器。
每次仅围绕一个独立决策添加
```
group.experiment()
```
。
在评分证明新路径更优之前，保持旧的提示词/模型/工具行为稳定。

当仓库中有多个候选工作流时，先使用

inngest-brownfield-audit

。当工作内容为AgentKit或持久化代理工作流时，结合

inngest-agents

使用本技能。

Anti-Patterns

反模式

Starting with a prompt experiment before naming the outcome metric.
Scoring with unstable names, strings, objects,
```
NaN
```
, or
```
Infinity
```
.
Forgetting
```
scoreMiddleware()
```
and then assuming
```
step.score()
```
is missing.
Running scorers in an external queue when the scorer needs durable waits.
Keeping experiment assignment only in memory.
Scoring an experiment variant from a later run without the original
```
runId
```
.
Using sessions for low-cardinality labels or sensitive personal data.
Leaving model/tool work outside steps, making traces unable to explain the score.

在确定结果指标之前就启动提示词实验。
使用不稳定的名称、字符串、对象、
```
NaN
```
或
```
Infinity
```
进行评分。
忘记添加
```
scoreMiddleware()
```
却误以为
```
step.score()
```
缺失。
当评分器需要持久化等待时，在外部队列中运行评分器。
仅在内存中保存实验分配信息。
在后续运行实例中为实验变体评分时，未提供原始
```
runId
```
。
将会话用于低基数标签或敏感个人数据。
将模型/工具工作放在步骤之外，导致追踪记录无法解释评分原因。

Verification

验证

Typecheck the Inngest client, functions, scorer schemas, and serve handler.
Confirm scorers are registered with the serve endpoint.
Test the producer emits events with expected
```
meta.sessions
```
.
Test direct scores use stable names and finite number/boolean values.
Test deferred scorer timeout behavior.
For experiments, test that each variant callback contains step work and that later scoring receives
```
experimentRef
```
plus the original run ID when needed.
If possible, run the Inngest dev server and inspect traces, sessions, scores, and experiment variant selection in the dashboard or through available CLI/API tooling.

对Inngest客户端、函数、评分器模式和服务处理程序进行类型检查。
确认评分器已在服务端点注册。
测试生产者是否按预期发送带有
```
meta.sessions
```
的事件。
测试直接评分使用稳定名称和有限数字/布尔值。
测试延迟评分器的超时行为。
对于实验，测试每个变体回调包含步骤工作，且后续评分在需要时能获取
```
experimentRef
```
和原始运行ID。
如有可能，运行Inngest开发服务器，并通过仪表板或可用的CLI/API工具检查追踪记录、会话、评分和实验变体选择情况。