write-judge-prompt
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseWrite LLM-as-Judge Prompt
编写LLM-as-Judge提示词
Design a binary Pass/Fail LLM-as-Judge evaluator for one specific failure mode. Each judge checks exactly one thing.
为某一特定故障模式设计一个二元判定（Pass/Fail）的LLM-as-Judge评估器。每个评判器仅负责检查一项内容。
Prerequisites
前提条件
- Error analysis is complete. The failure mode is identified.
- You have human-labeled traces for this failure mode (at least 20 Pass and 20 Fail examples).
- A code-based evaluator cannot check this failure mode. Exhaust code-based options before reaching for a judge — many failure modes that seem subjective reduce to keyword checks, regex, or API calls when you understand the domain. Example: detecting whether an AI interviewing coach suggests "general" questions (asking about typical behavior instead of a specific past event) seems to require semantic understanding, but in practice a keyword check for words like "usually," "typical," and "normally" could work quite well.
- 已完成错误分析，明确故障模式。
- 拥有该故障模式的人工标注样本（至少20个Pass样本和20个Fail样本）。
- 代码式评估器无法检查该故障模式。在选择使用评判器前，请先穷尽代码式方案——许多看似主观的故障模式，在深入理解领域后，可简化为关键词检查、正则表达式或API调用。例如：检测AI面试教练是否建议“通用”问题（询问典型行为而非特定过往事件）看似需要语义理解，但实际上通过检查“usually”“typical”“normally”等关键词就能达到不错的效果。
The Four Components
四个核心组件
Every judge prompt requires exactly four components:
每个评判器提示词都必须包含以下四个组件：
1. Task and Evaluation Criterion
1. 任务与评估标准
State what the judge evaluates. One failure mode per judge.
You are an evaluator assessing whether a real estate assistant's email
uses the appropriate tone for the client's persona.Not: "Evaluate whether the email is good" or "Rate the email quality from 1-5."
明确评判器的评估内容。每个评判器对应一种故障模式。
You are an evaluator assessing whether a real estate assistant's email
uses the appropriate tone for the client's persona.错误示例：“评估这封邮件是否优秀”或“从1-5分对邮件质量打分”。
2. Pass/Fail Definitions
2. Pass/Fail定义
Outcomes are strictly binary: Pass or Fail. No Likert scales, no letter grades, no partial credit. Define exactly what constitutes Pass and Fail. These definitions come from your error analysis failure mode descriptions.
undefined结果严格为二元：Pass或Fail。不使用李克特量表、字母等级或部分得分。明确定义Pass和Fail的具体判定标准，这些定义源自错误分析中的故障模式描述。
undefinedDefinitions
Definitions
PASS: The email matches the expected communication style for the client persona:
- Luxury Buyers: formal language, emphasis on exclusive features, premium market positioning, no casual slang
- First-Time Homebuyers: warm and encouraging tone, educational explanations, avoids jargon, patient and supportive
- Investors: data-driven language, ROI-focused, market analytics, concise and professional
FAIL: The email uses a tone mismatched to the client persona. Examples:
- Using casual slang ("hey, check out this pad!") for a luxury buyer
- Using heavy financial jargon for a first-time homebuyer
- Using overly emotional language for an investor
undefinedPASS: The email matches the expected communication style for the client persona:
- Luxury Buyers: formal language, emphasis on exclusive features, premium market positioning, no casual slang
- First-Time Homebuyers: warm and encouraging tone, educational explanations, avoids jargon, patient and supportive
- Investors: data-driven language, ROI-focused, market analytics, concise and professional
FAIL: The email uses a tone mismatched to the client persona. Examples:
- Using casual slang ("hey, check out this pad!") for a luxury buyer
- Using heavy financial jargon for a first-time homebuyer
- Using overly emotional language for an investor
undefined3. Few-Shot Examples
3. 少样本示例
Include labeled Pass and Fail examples from your human-labeled data.
undefined从人工标注数据中选取已标注的Pass和Fail示例。
undefinedExamples
Examples
Example 1: PASS
Example 1: PASS
Client Persona: Luxury Buyer
Email: "Dear Mr. Harrington, I am pleased to present an exclusive listing
at 1200 Pacific Heights Drive. This distinguished property features..."
Critique: The email opens with a formal salutation and uses language
consistent with luxury positioning — "exclusive listing," "distinguished
property." No casual slang or informal phrasing. The tone matches the
luxury buyer persona throughout.
Result: Pass
Client Persona: Luxury Buyer
Email: "Dear Mr. Harrington, I am pleased to present an exclusive listing
at 1200 Pacific Heights Drive. This distinguished property features..."
Critique: The email opens with a formal salutation and uses language
consistent with luxury positioning — "exclusive listing," "distinguished
property." No casual slang or informal phrasing. The tone matches the
luxury buyer persona throughout.
Result: Pass
Example 2: FAIL
Example 2: FAIL
Client Persona: Luxury Buyer
Email: "Hey! Just found this awesome place you might like. It's got a
pool and stuff, super cool neighborhood..."
Critique: The greeting "Hey!" is informal. Phrases like "awesome place,"
"got a pool and stuff," and "super cool" are casual slang inappropriate
for a luxury buyer. The email reads like a text message, not a
professional communication for a high-end client.
Result: Fail
Client Persona: Luxury Buyer
Email: "Hey! Just found this awesome place you might like. It's got a
pool and stuff, super cool neighborhood..."
Critique: The greeting "Hey!" is informal. Phrases like "awesome place,"
"got a pool and stuff," and "super cool" are casual slang inappropriate
for a luxury buyer. The email reads like a text message, not a
professional communication for a high-end client.
Result: Fail
Example 3: PASS (borderline)
Example 3: PASS (borderline)
Client Persona: First-Time Homebuyer
Email: "Hi Sarah, I found a property that might be a great fit for your
first home. The neighborhood has good schools nearby, and the monthly
payment would be similar to what you're currently paying in rent..."
Critique: The greeting is warm but not overly casual. The email explains
the property in relatable terms — comparing mortgage to rent, mentioning
schools — which is educational without being condescending. It avoids
jargon like "amortization" or "LTV ratio." While not deeply technical,
this matches the supportive tone expected for a first-time buyer.
Result: Pass
**Rules for selecting examples:**
- Include at least one clear Pass, one clear Fail, and one borderline case. Borderline examples are the most valuable — they teach nuance.
- Draw examples from the training split (10-20% of labeled data set aside for this purpose).
- Any example used in the judge prompt must be excluded from dev and test sets. Using dev/test examples is data leakage.
- 2-4 examples is typical. Performance plateaus after 4-8.Client Persona: First-Time Homebuyer
Email: "Hi Sarah, I found a property that might be a great fit for your
first home. The neighborhood has good schools nearby, and the monthly
payment would be similar to what you're currently paying in rent..."
Critique: The greeting is warm but not overly casual. The email explains
the property in relatable terms — comparing mortgage to rent, mentioning
schools — which is educational without being condescending. It avoids
jargon like "amortization" or "LTV ratio." While not deeply technical,
this matches the supportive tone expected for a first-time buyer.
Result: Pass
**示例选择规则：**
- 至少包含1个明确的Pass示例、1个明确的Fail示例和1个边界案例。边界案例最具价值——它们能传递评判的细微差别。
- 从训练集中选取示例（标注数据中预留10-20%用于此目的）。
- 提示词中使用的任何示例都必须排除在开发集和测试集之外，否则会导致数据泄露。
- 通常使用2-4个示例，超过4-8个后性能会趋于平稳。4. Structured Output Format
4. 结构化输出格式
Enforce structured output using your LLM provider's schema enforcement (e.g., in OpenAI, tool definitions in Anthropic) or a library like Instructor or Outlines. If the provider doesn't support schema enforcement, specify the JSON schema in the prompt.
response_formatThe output must include a critique before the verdict. Placing the critique first forces the judge to articulate its assessment before committing to a decision.
json
{
"critique": "string — detailed assessment of the output against the criterion",
"result": "Pass or Fail"
}Critiques must be detailed, not terse. A good critique explains what specifically was correct or incorrect and references concrete evidence from the output. The critiques in your few-shot examples set the bar for the level of detail the judge will produce.
利用LLM提供商的模式强制执行功能（如OpenAI的、Anthropic的工具定义）或Instructor、Outlines等库来强制结构化输出。如果提供商不支持模式强制执行，则在提示词中指定JSON schema。
response_format输出必须在verdict（判定结果）前包含critique（评估意见）。将评估意见放在前面，可迫使评判器在做出决定前先阐明其评估依据。
json
{
"critique": "string — detailed assessment of the output against the criterion",
"result": "Pass or Fail"
}评估意见必须详细，不能过于简洁。优质的评估意见应明确指出输出内容中哪些部分正确或错误，并引用输出中的具体证据。少样本示例中的评估意见为评判器应达到的详细程度设定了标准。
Choosing What to Pass to the Judge
选择传递给评判器的信息
Feed only what the judge needs for an accurate decision:
| Failure Mode | What the Judge Needs |
|---|---|
| Tone mismatch | Client persona + generated email |
| Answer faithfulness | Retrieved context + generated answer |
| SQL correctness | User query + generated SQL + schema |
| Instruction following | System prompt rules + generated response |
| Tool call justification | Conversation history + tool call + tool result |
For long documents, feed only the relevant snippet, not the entire document.
仅提供评判器做出准确决策所需的信息：
| 故障模式 | 评判器所需信息 |
|---|---|
| 语气不符 | 客户角色 + 生成的邮件 |
| 回答忠实度 | 检索到的上下文 + 生成的回答 |
| SQL正确性 | 用户查询 + 生成的SQL + schema |
| 指令遵循度 | 系统提示规则 + 生成的响应 |
| 工具调用合理性 | 对话历史 + 工具调用 + 工具结果 |
对于长文档，仅提供相关片段，而非完整文档。
Model Selection
模型选择
Start with the most capable model available. The same model used for the main task works as judge (the judge performs a different, narrower task). Optimize for cost later once alignment is confirmed.
从可用的最强大模型开始。用于主任务的模型也可作为评判器（评判器执行的是不同的、更窄范围的任务）。在确认对齐后，再针对成本进行优化。
Anti-Patterns
反模式
- Vague criteria like "is this helpful?" Target a specific, observable failure mode from error analysis.
- Holistic judge for the entire trace. A single judge covering multiple dimensions produces unactionable verdicts.
- No few-shot examples. Without examples, the model won't know what counts as a failure in your application.
- Dev/test examples used as few-shot. This is data leakage. Use only the training split.
- Likert scales (1-5, letter grades, etc.). Binary pass/fail only. Likert scales produce scores that sound precise but can't be calibrated: annotators disagree on the difference between a 3 and a 4, and the judge inherits that noise. Binary forces you to define a clear decision boundary upfront, which makes inter-annotator agreement measurable and the judge's errors actionable. If you need to capture severity, use multiple binary judges (e.g., "factually wrong" and "dangerously wrong") rather than one ordinal scale.
- Skipping validation. Measure alignment with human labels using validate-evaluator before trusting the judge.
- Judges for specification failures without fixing the prompt first. If the prompt never asked for the behavior, add the instruction before building an evaluator. For critical requirements, a judge can still serve as a regression guard.
- 模糊标准，如“这是否有用？” 目标应是错误分析中明确的、可观察的故障模式。
- 针对整个轨迹的整体评判器 单个评判器涵盖多个维度会产生无法落地的判定结果。
- 缺少少样本示例 没有示例的话，模型无法了解您的应用中什么情况属于故障。
- 将开发/测试集示例用作少样本 这属于数据泄露，仅使用训练集。
- 李克特量表（1-5分、字母等级等） 仅使用二元Pass/Fail判定。李克特量表产生的分数看似精确，但无法校准：标注者对3分和4分的差异意见不一，评判器会继承这种噪声。二元判定要求您预先明确清晰的决策边界，这使得标注者间的一致性可衡量，且评判器的错误可落地解决。如果需要区分严重程度，请使用多个二元评判器（如“事实错误”和“危险错误”），而非单一的序数量表。
- 跳过验证 在信任评判器之前，请使用validate-evaluator衡量其与人工标注的对齐程度。
- 在未修复提示词前就针对规范故障使用评判器 如果提示词从未要求某种行为，请先添加相关指令，再构建评估器。对于关键要求，评判器仍可作为回归防护手段。