Back to Details

langsmith-online-eval-engineering

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Online Eval Engineering

在线评估工程

Build online evaluators iteratively:
text
inspect traces and interview user -> propose directions -> user chooses
-> build evaluator -> test, attach, verify -> review and repeat
Read references/langsmith-api.md before creating or modifying evaluators.
迭代式构建在线评估器：
text
检查追踪记录并与用户沟通 -> 提出方向 -> 用户选择
-> 构建评估器 -> 测试、关联、验证 -> 复盘并重复
在创建或修改评估器前，请阅读references/langsmith-api.md。

1. Inspect traces

1. 检查追踪记录

Ask the user for their LangSmith project name. Fetch recent root-level traces and print their structure. Read references/trace-inspection.md. Find:
  • run name and type;
  • available input and output field names;
  • the shape and content of the data (truncated samples);
  • which fields carry the data an evaluator would need.
Summarize the trace structure in the conversation:
text
Project: name
Run type: chain | llm | tool | ...
Input fields: field names and what they contain
Output fields: field names and what they contain
Sample: one representative input/output pair (truncated)
Keep the user involved: explain the trace structure and what it implies, then ask only for information the traces cannot establish. For example: "What does this application do?", "What quality concern matters most?", or "What failure should never happen?"
Ask whether the user wants a naming prefix for evaluators in this session (e.g., 
myapp-
, 
v2-
, 
dogfood-
). If they provide one, apply it to all evaluator names, prompt hub handles, and run rule display names. If they decline, use plain descriptive names.
Do not propose evaluators until the trace structure is understood and the user has described their concerns.
向用户询问其LangSmith项目名称。获取最近的顶级追踪记录并打印其结构。阅读references/trace-inspection.md，找出：
  • 运行名称与类型；
  • 可用的输入和输出字段名称；
  • 数据的结构与内容（截断样本）；
  • 评估器所需数据所在的字段。
在对话中总结追踪记录结构：
text
项目：名称
运行类型：chain | llm | tool | ...
输入字段：字段名称及其包含内容
输出字段：字段名称及其包含内容
样本：一组有代表性的输入/输出对（已截断）
让用户全程参与：解释追踪记录结构及其含义，仅询问追踪记录无法体现的信息。例如：“这个应用的功能是什么？”、“最需要关注的质量问题是什么？”或“绝对不能出现哪种故障？”
询问用户是否希望本次会话中的评估器使用命名前缀（如
myapp-
、
v2-
、
dogfood-
）。若用户提供前缀，将其应用于所有评估器名称、Prompt Hub句柄和运行规则显示名称。若用户拒绝，则使用简洁的描述性名称。
在理解追踪记录结构且用户说明其关注点之前，不要提出评估器方案。

2. Discuss and choose an eval direction

2. 讨论并选择评估方向

Read references/evaluator-design.md. Propose two or three evaluation criteria grounded in the trace data. Apply the naming prefix from step 1 if the user provided one. For each, give:
text
Name: descriptive evaluator name (with prefix if set)
Type: LLM-as-judge or code
Measures: what quality dimension this evaluates
Scoring: bool, float (0-1), or int; what pass/fail means
Fields needed: which trace fields are used and how
Rationale: why this type and approach
Example:
text
Name: response-relevance
Type: LLM-as-judge
Measures: whether the response addresses the user's question
Scoring: bool; True = relevant, False = off-topic or non-responsive
Fields needed: input (user question), output (assistant response)
Rationale: relevance is semantic and requires reading comprehension; not decidable by code
Recommend one and ask the user which to build. Do not implement until the user chooses.
阅读references/evaluator-design.md。基于追踪数据提出两到三个评估标准。若用户在步骤1中提供了前缀，需应用该前缀。针对每个标准，提供以下信息：
text
名称：描述性评估器名称（若设置前缀则包含）
类型：LLM-as-judge 或代码
评估维度：该评估器衡量的质量维度
评分方式：bool、float（0-1）或int；说明合格/不合格的判定规则
所需字段：使用的追踪字段及使用方式
理由：选择该类型与方法的原因
示例：
text
名称：response-relevance
类型：LLM-as-judge
评估维度：响应是否回应用户问题
评分方式：bool；True=相关，False=偏离主题或无响应
所需字段：input（用户问题）、output（助手响应）
理由：相关性属于语义范畴，需要阅读理解能力，无法通过代码判定
推荐其中一个方案，并询问用户选择构建哪一个。在用户做出选择前不要开始实现。

3. Build one evaluator

3. 构建单个评估器

Read references/langsmith-api.md. Build the selected evaluator. Show the full configuration to the user and get approval before executing any API calls.
LLM-as-judge path. Define a 
ResponseSchema
with 
reasoning
first, then the score field. Write prompt messages with a clear rubric that assesses the result, not whether it matches a reference answer. Set 
variable_mapping
using field names discovered in step 1. Present the schema, prompt, variable mapping, and evaluator name for approval. On approval, push the prompt and create the evaluator. Report the evaluator ID.
Code evaluator path. Write a 
perform_eval(run, example=None)
function. It must be self-contained (only builtins and standard library), access 
run
as a dict (
run.get("outputs")
), and return 
{"key": ..., "score": ..., "comment": ...}
. Present the function code and evaluator name for approval. On approval, create the evaluator. Report the evaluator ID.
阅读references/langsmith-api.md。构建用户选中的评估器。在执行任何API调用前，向用户展示完整配置并获得批准。
LLM-as-judge路径。定义一个
ResponseSchema
，先包含
reasoning
字段，再包含评分字段。编写带有明确评估标准的提示消息，评估结果而非是否匹配参考答案。使用步骤1中发现的字段名称设置
variable_mapping
。向用户展示Schema、提示词、变量映射和评估器名称以获得批准。批准后，推送提示词并创建评估器，报告评估器ID。
代码评估器路径。编写
perform_eval(run, example=None)
函数。函数必须独立可运行（仅使用内置函数和标准库），以字典形式访问
run
（如
run.get("outputs")
），并返回
{"key": ..., "score": ..., "comment": ...}
。向用户展示函数代码和评估器名称以获得批准。批准后，创建评估器，报告评估器ID。

4. Test, attach, and verify

4. 测试、关联与验证

Before attaching, ask the user what sampling rate they want (1.0 = every trace, 0.5 = half, 0.1 = 10%, or custom). Do not default silently. If the user is unsure, recommend 1.0 for initial testing.
Ask the user whether they want to test the evaluator against a few existing traces before attaching. Run rules only fire on new traces, so historical testing is the only way to verify before new traffic arrives.
For code evaluators, execute 
perform_eval
directly against fetched root-level traces, passing a dict with 
inputs
, 
outputs
, and 
attachments
keys. This catches runtime errors (wrong field names, dict-vs-object access, missing data) before production. For LLM evaluators, verify the configuration: confirm 
variable_mapping
keys match prompt placeholders, confirm mapped trace fields exist, and check that the mapped data is meaningful.
If testing reveals errors, fix and recreate before attaching. If the user declines testing, proceed to attach.
Create a run rule to connect the evaluator to the tracing project. Apply the user's naming prefix to the 
display_name
. Confirm the evaluator appears in the evaluator list with the correct project attachment. Inspect:
  • evaluator attachment and run rule status;
  • recent trace feedback and scores (from historical testing or new traces);
  • whether scores match expectations for the traces inspected;
  • edge case handling (empty output, errored runs, unexpected structure).
Fix and reattach when the evaluator crashes, scores incorrectly, or fails on edge cases. Before approval, confirm the evaluator scored the intended quality dimension, not an infrastructure or data-shape failure.
关联之前，询问用户希望使用的采样率（1.0=所有追踪记录，0.5=一半，0.1=10%，或自定义）。不要默认设置。若用户不确定，建议初始测试使用1.0。
询问用户是否希望先针对部分现有追踪记录测试评估器，再进行关联。运行规则仅对新追踪记录生效，因此历史测试是新流量到来前验证的唯一方式。
对于代码评估器，直接针对获取的顶级追踪记录执行
perform_eval
，传入包含
inputs
、
outputs
和
attachments
键的字典。这可以在上线前捕获运行时错误（如字段名称错误、字典与对象访问混淆、数据缺失）。对于LLM评估器，验证配置：确认
variable_mapping
键与提示词占位符匹配，确认映射的追踪字段存在，并检查映射数据是否有意义。
若测试发现错误，修复并重新创建评估器后再关联。若用户拒绝测试，则直接进行关联。
创建运行规则，将评估器与追踪项目关联。将用户提供的命名前缀应用于
display_name
。确认评估器出现在评估器列表中且已正确关联到项目。检查：
  • 评估器关联状态与运行规则状态；
  • 最近的追踪记录反馈与评分（来自历史测试或新追踪记录）；
  • 评分是否与检查过的追踪记录预期一致；
  • 边缘情况处理（空输出、运行错误、意外结构）。
若评估器崩溃、评分错误或无法处理边缘情况，需修复并重新关联。获得批准前，确认评估器针对的是预期的质量维度，而非基础设施或数据结构故障。

5. Review with the user

5. 与用户复盘

Explain the evaluator name and ID, quality dimension and scoring approach, trace fields used, sampling rate, and any limitation. Ask the user to approve, revise, drop, or choose the next direction. If continuing, reuse the trace findings, then propose a distinct quality dimension.
解释评估器的名称与ID、质量维度与评分方式、使用的追踪字段、采样率及任何局限性。询问用户是否批准、修改、放弃或选择下一个方向。若继续，复用之前的追踪记录发现，然后提出一个不同的质量维度。

Invariants

不变规则

  • One quality dimension per evaluator.
  • No guessing field names; always inspect traces before implementing.
  • Show configuration and get user approval before making API calls.
  • Code evaluators must be self-contained: only builtins and standard library.
  • Code evaluators receive 
    run
    as a plain dict; use 
    run.get("inputs")
    and 
    run.get("outputs")
    , not attribute access. The 
    example
    parameter must default to 
    None
    .
  • Treat API failures, auth errors, and run rule failures as infrastructure errors, not evaluator bugs.
  • 每个评估器仅对应一个质量维度。
  • 不要猜测字段名称；实现前必须检查追踪记录。
  • 在调用API前展示配置并获得用户批准。
  • 代码评估器必须独立可运行：仅使用内置函数和标准库。
  • 代码评估器接收的
    run
    为普通字典；使用
    run.get("inputs")
    和
    run.get("outputs")
    ，而非属性访问。
    example
    参数必须默认为
    None
    。
  • 将API失败、认证错误和运行规则失败视为基础设施错误，而非评估器bug。",