1. If the user passed

   --backend pup

   anywhere in their invocation → use pup mode immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether

   mcp__datadog-llmo-mcp__list_llmobs_evals

   appears in your available tools.
3. If MCP tools are present → use MCP mode throughout. Call MCP tools exactly as named in this skill's workflow sections.
4. If MCP tools are absent → check whether

   pup

   is executable: run

   pup --version

   via Bash. A JSON response containing

   "version"

   confirms pup is available.
5. If pup responds → use pup mode throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
6. If neither is available → stop and tell the user:

   "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (

   claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs'

   ) or install pup."

--backend pup

• Invoke via Bash:

  pup llm-obs <subcommand> [flags]

• pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in

  [{"type": "text", "text": "<json>"}]

  ).
• If pup returns an auth error, tell the user to run

  pup auth login

  and stop.
• Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).
• Time flags: pup accepts bare duration strings (

  1h

  ,

  7d

  ,

  30m

  ) and RFC3339 timestamps. Do not use

  now-

  -prefixed strings — strip the prefix when converting from a skill

  --timeframe

  argument:

  now-7d

  →

  7d

  ,

  now-24h

  →

  24h

  ,

  now-30d

  →

  30d

  .

• --summary

  on

  pup llm-obs spans search

  strips payload fields to essential metadata only. Use it in bulk/search phases where content is not needed.

3a9f1c2b

telemetry.intent

skill:llm-obs-eval-bootstrap[<inv_id>] — 

skill:llm-obs-eval-bootstrap:start[<inv_id>] — 

:start

skill:llm-obs-eval-bootstrap:start[3a9f1c2b] — Phase 0: map existing eval coverage for task-cruncher

• sdk_code

  (default) — Python

  .py

  file using the Datadog Evals SDK (

  BaseEvaluator

  /

  LLMJudge

  ) for offline experiments.

• data_only

  — self-contained JSON spec, framework-agnostic.

• publish

  — write online LLM-judge evaluators directly to Datadog via

  create_or_update_llmobs_evaluator

  . These run automatically on matching production spans or traces (no dataset, no task function). The skill auto-classifies each proposed evaluator as span-scoped or trace-scoped based on what the judgment requires (a per-LLM-call tone check vs. an agent goal completion that needs the whole trace) — the user accepts or overrides the classification at the proposal checkpoint.

1. get_llmobs_span_details

   : Group span_ids by trace_id. One call per trace_id with ALL its span_ids. Issue ALL calls for a page in a single message.

2. get_llmobs_span_content

   : Each call is independent — always issue ALL in a single message.

3. get_llmobs_trace

   /

   get_llmobs_agent_loop

   : Parallelize across different traces in a single message.
4. Pipeline parallelism: Start

   get_llmobs_span_details

   for page 1 results immediately — don't wait to collect all pages.

Applies to

sdk_code

mode only. In

data_only

mode, use this section as domain context when writing rubric prompts — no SDK classes are emitted.

structured_output

• system_prompt: Set the judge's role and the app's domain context. Does NOT support template vars.
• user_prompt: Present the data via

  {{input_data}}

  /

  {{output_data}}

  , then describe what good vs. bad looks like for this dimension.

• ddtrace.llmobs._evaluators.llm_judge

  —

  LLMJudge

  ,

  BooleanStructuredOutput

  ,

  ScoreStructuredOutput

  ,

  CategoricalStructuredOutput

• ddtrace.llmobs._experiment

  —

  BaseEvaluator

  ,

  EvaluatorContext

  ,

  EvaluatorResult

• ddtrace.llmobs._evaluators.format

  —

  JSONEvaluator

  ,

  LengthEvaluator

• ddtrace.llmobs._evaluators.string_matching

  —

  StringCheckEvaluator

  ,

  RegexMatchEvaluator

ml_app

ml_app

--timeframe

now-7d

--data-only

--publish

output_mode = publish

--publish

output_mode = data_only

--data-only

output_mode = sdk_code

--data-only

--publish

1. If

   ml_app

   not provided → ask the user.
2. Auto-detect entry mode:
   • If the conversation contains an RCA report (look for "Failure Taxonomy" heading, structured failure modes, or severity ratings) →

     from_rca

     . Extract the taxonomy.
   • If the user provides a free-text failure hypothesis (e.g., "the system prompt lacks grounding") →

     from_rca

     . Use the hypothesis as the starting eval target.
   • Otherwise →

     cold_start

     .
3. If

   timeframe

   not provided → default to

   now-7d

   .
4. Map existing eval coverage — skip if

   output_mode = data_only

   (there is no Datadog eval project to check coverage against): Call

   list_llmobs_evals

   (org-wide; filter the result client-side to entries where

   ml_app == <ml_app>

   ). Then, for each eval with

   source=custom

   , call

   get_llmobs_evaluator(eval_name=...)

   to inspect its prompt template, target, sampling, and filter, and infer which quality dimension it covers. Issue all evaluator calls in a single message (parallelize). Skip

   source=ootb

   evals — their names are self-describing and they may not have a fetchable config.
   By the end of this step you have a complete coverage map:

   {eval_name → source, enabled, dimension}

   . Carry this into Phase 2 for deduplication.
   In

   publish

   mode, also note any template-variable convention the existing custom evaluators already use (so a new suite reads consistently). Online evaluator templates resolve against the full span JSON, not against

   EvaluatorContext

   . See the "Online Template Variables" section under "Publishing Conventions" for the supported syntax (

   {{span_input}}

   ,

   {{span_output}}

   , dot-paths, array selectors, filter accessors).
5. Notebook context detection: Scan the current conversation for a Datadog notebook URL that was produced by

   /eval-trace-rca

   (pattern:

   https://app.datadoghq.com/notebook/{numeric-id}

   ). If found, store it as

   rca_notebook_url

   and extract the numeric ID as

   rca_notebook_id

   . This is used after Phase 3 to offer appending the evaluator suite to that notebook instead of creating a new one.

1. Sample the app:

   search_llmobs_spans(query="@ml_app:\"<ml_app>\" @status:ok", root_spans_only=true, limit=50, from=<timeframe>)

   . Filter by

   @status:ok

   — error spans have no output to evaluate.
2. Profile the app and identify evaluation target spans: Call

   get_llmobs_span_details

   for span_ids grouped by trace_id. Inspect

   content_info

   to classify:

   Signal	App Profile
   content_info has messages	LLM/chat app
   content_info has documents	RAG app
   Spans include agent kind	Agent app
   content_info has metadata	Has custom metadata
   Multiple span kinds in one trace ( agent + tool / retrieval + llm from get_llmobs_trace )	Multi-step app — at least one trace-scope evaluator likely belongs in the suite ( publish mode)

   For agent/multi-step apps, also call

   get_llmobs_trace

   on 2-3 traces to see the full span hierarchy. Compare

   content_info

   between the root span and its sub-spans. Then ask two questions for each candidate quality dimension, in this order:
   1. Does the verdict depend on more than one span? (e.g., faithfulness depends on a

      retrieval

      span's documents AND an

      llm

      span's answer; goal completion depends on the chain of

      tool

      calls AND the final response.) If yes → trace scope in

      publish

      mode. Don't try to compress this into a single span.
   2. Only if the answer to (1) is no: pick the single span with the richest signal for that dimension (root has the summary; LLM sub-spans have the full system prompt + tool call results + reasoning chain).
   Record the span-kind histogram (agent + tool + llm + retrieval) — multiple kinds under one root is a strong signal you'll have at least one trace-scope evaluator in the suite. See Phase 2's "Span vs. Trace Scope Classification" for the mandatory walk-through of canonical trace-scope use cases.
3. Extract content and identify targets: Call

   get_llmobs_span_content

   for representative spans. Fetch fields based on app profile:

   App Profile	Fields to Fetch
   LLM/chat	messages ( path=$.messages[0] for system prompt), output
   RAG	documents , input , output
   Agent	get_llmobs_agent_loop for the agent span, then messages for detail
   Any with metadata	metadata

   Issue all calls in a single message. As you read, capture two streams of signal:
   Generic quality signals — what does "success" look like? What variance exists across outputs? Each observed quality dimension becomes a candidate evaluator, with the traces you've just read as evidence. Also look for safety signals (scope violations, sensitive data in outputs, out-of-character responses) and add a safety evaluator if you find them.
   Domain signals — these become the domain-specific evaluator category in Phase 2 (the highest-leverage category). For every 5–10 traces, write down:
   • Recurring intents / question categories — what classes of request does this app handle? (

     applying for benefit X

     ,

     comparing flight options

     ,

     summarizing a policy

     ,

     creating a widget

     )
   • Entities the app emits in outputs — URLs, agency / company names, code identifiers, monetary amounts, dates, IDs, file paths, phone numbers. Note which ones the user acts on downstream (those are worth a correctness evaluator) versus which are passing references.
   • Tool argument shapes (for agent apps) — name each tool the agent calls and the rough schema of its inputs. Tools with non-trivial schemas (≥ 3 fields, structured types) are candidates for argument-correctness evaluators.
   • Persona / voice rules — does the app always cite a source, always refuse certain topics (medical, legal, financial advice), always speak in a particular tone? Extract the rules implicitly followed across observed outputs.
   • Failure modes specific to the domain — fabricated identifiers, outdated policy references, currency / locale mismatches, off-by-one errors in IDs, wrong units. One observed instance is enough to seed a candidate evaluator.
   Don't try to enumerate domain signals exhaustively before reading traces — let the patterns surface as you read. The goal is breadth in the eventual proposal, not completeness in this exploration step.

1. Extract the failure taxonomy from the RCA report. Each failure mode with High or Medium severity becomes an eval target.
2. Check root cause categories for infrastructure failures. Before proposing evaluators, scan the Root Cause column of the taxonomy for any of:

   Instrumentation Deficiency

   ,

   Harness Deficiency

   ,

   Runtime Error

   ,

   Upstream Data Issue

   , or any other root cause that points to infrastructure/environment rather than model behavior. If any are present, pause and ask:

   "Some failure modes were diagnosed as infrastructure or instrumentation issues rather than model behavior (e.g.,

   {list the infra root causes}

   ). Evaluators can be designed two ways:

   • Behavior-targeted (recommended for ongoing quality): measure whether the model produces correct, specific output — useful once the infrastructure is fixed and you want to track real quality
   • Artifact-targeted (useful as regression guard): detect the specific broken output observed (e.g., generic placeholder responses) — catches regressions if the infrastructure breaks again

   Which approach do you want, or both?"

   • If behavior-targeted: design evaluators for what correct output looks like, not what the broken output looked like. Use the RCA's

     expected_output

     / gold-standard examples as the quality bar.
   • If artifact-targeted: design evaluators that detect the specific failure symptom (e.g.,

     StringCheckEvaluator

     for a known bad string,

     LLMJudge

     that checks for generic placeholders).
   • If both: propose each category separately, clearly labelled.
   If all root causes are behavioral (System Prompt Deficiency, Tool Gap, Tool Misuse, Retrieval Failure, etc.) → skip this step and proceed directly.
3. For each target: if the RCA includes trace IDs, use them directly; otherwise search for matching traces. Fetch 2-3 traces per target with

   get_llmobs_span_content

   to understand the concrete pattern.

output_mode

• sdk_code

  /

  data_only

  → offline experiments. Template variables use

  EvaluatorContext

  fields (

  {{input_data}}

  ,

  {{output_data}}

  ). The actual data shape depends on the user's dataset and task function (see EvaluatorContext note in SDK Reference).

• publish

  → online evaluation on production spans. Template variables resolve against the full span JSON via dot-paths (

  {{meta.input.value}}

  ,

  {{meta.output.messages[*].content}}

  , …) or the built-in span-kind-aware aliases (

  {{span_input}}

  ,

  {{span_output}}

  ). See "Online Template Variables" under Publishing Conventions for the full syntax. Each evaluator also needs

  eval_scope

  ,

  sampling_percentage

  , and (optionally)

  filter

  — surface these in the proposal table so the user can confirm before publishing.

1. Domain-specific evaluators — What does "good" mean for this specific app? These are the highest-leverage proposals because they capture quality bars generic evaluators miss. Derive them from the domain signals Phase 1 captured:
   • Recurring intents / question categories the app handles (e.g., "applying for a federal benefit", "comparing flight options", "explaining a policy"). Propose an

     intent_classification

     or

     intent_handling_correctness

     evaluator scoped to the dominant intents.
   • Specific entities the app produces (URLs, agency names, code identifiers, monetary amounts, dates, IDs). Propose a per-entity correctness evaluator for the ones with real downstream cost when wrong (e.g.,

     cited_url_is_real

     ,

     agency_name_matches_request

     ,

     monetary_amount_is_consistent_with_input

     ).
   • Tool argument shapes observed across

     tool

     spans. Propose a per-tool argument-correctness evaluator for the tools with non-trivial schemas (e.g.,

     search_flights_args_match_user_request

     ,

     update_dashboard_widget_targets_correct_widget

     ).
   • Persona / voice expectations — does the app always cite sources, always refuse out-of-scope requests, always speak in a specific tone? Propose evaluators for the voice rules you can extract from observed outputs (

     cites_a_source

     ,

     refuses_medical_advice

     ,

     tone_matches_brand

     ).
   • Domain-specific failure modes seen across traces (fabricated identifiers, outdated policy references, unit mismatches, currency / locale mismatches). One evaluator per recurring failure mode.
   Name each evaluator after the user-facing concern, not the technical check (

   agency_url_is_real

   over

   regex_url_match

   ). Use the trace IDs you read in Phase 1 as evidence — at least one passing case and one failing case per evaluator if you saw both.
2. Outcome evaluators — Did this span / trace produce a good result for the request?
   • Examples:

     task_completion

     ,

     answer_correctness

     ,

     response_groundedness

3. Format evaluators — Does the output meet structural requirements?
   • Examples:

     valid_json_output

     ,

     response_length

     ,

     citation_format

4. Safety evaluators — Does the output stay within appropriate boundaries?
   • Examples:

     no_pii_leakage

     ,

     scope_adherence

     ,

     no_hallucination

4-6

• Aim for 8–15 evaluators in the proposal, distributed across all four categories (with domain-specific usually the largest bucket, outcome second, format and safety smaller). For very simple single-LLM-call apps, fewer is fine; for agent / RAG apps with rich domain signals, lean toward the upper end.
• Quality > generic: every domain-specific proposal should be backed by at least one observed pattern in the sampled traces. Don't invent generic domain evaluators ("

  response_quality

  ") if you don't have evidence for them.
• Let the user curate: the MANDATORY CHECKPOINT below explicitly asks the user to remove what doesn't apply, not just to approve. Treat the proposal as a candidate set the user trims.

data_only

(dimension, scope)

1. Enabled span-scope eval (OOTB or custom) for dimension D:
   • Do NOT propose a new span-scope evaluator for D — that dimension is already covered at span scope.
   • DO propose a trace-scope evaluator for D when the trace shape calls for it (multi-step app, judgment depends on cross-span context). Note the relationship in the rationale: e.g., "OOTB

     Goal Completeness

     evaluates each LLM span in isolation; this trace-scope

     goal_completion

     checks whether the agent's full sequence of steps achieved the user's request — different question."
2. Enabled trace-scope custom eval for dimension D: do NOT propose another trace-scope evaluator for the same dimension; that's a real duplicate. Span-scope on the same dimension is still fair game if the data also fits a single span.
3. Disabled OOTB eval: Do NOT propose a new custom span-scope evaluator for that dimension. Instead, surface it in a short note within the proposal and suggest enabling it in the Datadog UI rather than creating a duplicate. Example:

   hallucination

   (ootb, disabled) — consider enabling in Datadog UI (Evaluations → Configure) instead of creating a custom span-scope eval. (A trace-scope

   rag_faithfulness

   is still in scope and covers a different question.)

4. Gap identification: Open the proposal with a coverage summary line: "Existing coverage: N evaluator(s) already configured ({names}, all span-scope unless noted). Proposing evaluators for uncovered dimensions and uncovered scopes."
5. All dimensions covered: A dimension is "fully covered" only when both scopes are present (or the scope doesn't apply to the app shape). If the coverage map accounts for every identified quality dimension at the appropriate scope(s), surface this explicitly and ask the user what they want: (a) review/improve existing eval prompts, (b) add coverage for additional dimensions, or (c) proceed anyway.

• Name: Must match

  ^[a-zA-Z0-9_-]+$

  (alphanumeric, underscore, hyphen only)
• Type:

  LLMJudge

  (Boolean/Score/Categorical/custom JSON schema), built-in (

  JSONEvaluator

  ,

  RegexMatchEvaluator

  , etc.), or

  BaseEvaluator

  subclass. In

  publish

  mode, only LLM-judge evaluators are supported by the MCP tool — code-based checks must NOT be silently dropped. List them in the same proposal table with

  Type

  set to the code-based class, mark them under a "Not publishable in this mode" subsection of the proposal, and tell the user to run the skill again in default

  sdk_code

  mode (or

  --data-only

  ) to capture them. Treat the code-based proposals as part of the suite for counting and coverage purposes.
• What it measures: 1-2 sentence plain-language description
• Target span: Which span's data the evaluator was designed for (e.g., "root agent span", "LLM sub-span

  anthropic.request

  ", "all

  llm

  spans"). If the root span's I/O is too lossy for the quality dimension (e.g., tool call results aren't visible), note this and specify which sub-span has the signal. In

  publish

  mode this maps to a combination of

  eval_scope

  (

  span

  /

  trace

  /

  session

  ),

  root_spans_only

  , and the EVP

  filter

  query (e.g.

  @meta.span.kind:llm

  or

  service:web

  ).
• Pass/fail criteria:

  pass_when=True

  ,

  min_threshold=7

  ,

  pass_values=["correct"]

  , or "no automatic assessment" for custom JSON schema
• Template variables: Which of

  input_data

  ,

  output_data

  ,

  expected_output

  ,

  metadata.*

  it uses (offline) — or which span paths / aliases it pulls from (publish mode:

  {{span_input}}

  ,

  {{span_output}}

  ,

  {{meta.input.messages[*].content}}

  ,

  {{meta.metadata.<key>}}

  , etc.)
• Evidence: At least one trace where it would have caught a failure (or confirmed correct behavior)
• Publish-only fields (only in

  publish

  mode):

  integration_provider

  (default

  openai

  ),

  model_name

  (default

  gpt-5.4-mini

  ),

  sampling_percentage

  (default

  10

  ),

  eval_scope

  (default

  span

  ), and any

  filter

  query needed to scope to the right spans. Surface defaults in the proposal so the user can override before publishing.

• integration_account_id

  (only in

  publish

  mode): the integration account the judge LLM is called through. Auto-detected from existing evaluators in the same ml_app (Phase 0 coverage map). Never asked from the user as a raw UUID. If no existing evaluator has one, the field is omitted and the user picks an account in the UI before activating. All evaluators are published with

  enabled: false

  regardless — see "Always publish as draft" in Phase 3C for the full activation workflow.

tool

retrieval

workflow

agent

eval_scope: trace

goal_completion

tool_use_correctness

tool

rag_faithfulness

retrieval

conversation_quality

llm

1. Can the judgment be answered correctly from one span's

   meta.input

   +

   meta.output

   , where "correctly" means the verdict cannot change if you considered other spans in the trace? →

   eval_scope: span

   .
2. Otherwise →

   eval_scope: trace

   . In particular, default to trace when the evaluator name contains grounding, faithfulness, hallucination, completeness, correctness across steps, consistency, or workflow — these almost always need cross-span context.

Example rationales:

• tone_check

  — span. Judging "is this single response polite" needs only one LLM span's

  meta.output.messages[*].content

  ; no other span in the trace can change that verdict.

• goal_completion

  — trace. Whether the agent finished the user's request depends on the sequence of tool calls and the final LLM response together —

  meta.output

  of any single span only shows that step's output.

• tool_use_correctness

  — trace. Comparing tool inputs against the request and the final response requires correlating ≥ 3 spans (root, tool, final LLM).

• rag_faithfulness

  — trace. Grounding pairs the

  retrieval

  span's documents with the LLM span's answer.

Example "Skipped trace-scope candidates" entry:

• conversation_quality

  — skipped: traces contain a single LLM call (no multi-turn signal in this app's instrumentation).

publish

• {name}: {what it measures}
  • Target span: {which span's data it was designed for}
  • Rationale: {which quality dimension it covers and why}
  • {Only in publish mode:} Scope: {span | trace} — {one-sentence rationale}
  • Evidence: Trace {id_short}

• {canonical_use_case}

  — {one-line reason it does not apply to this app}

• {name}

  ({type}) — {what it would check}. Re-run

  /eval-bootstrap {ml_app}

  in default mode to emit as offline SDK code, or

  /eval-bootstrap {ml_app} --data-only

  for a framework-agnostic JSON spec.

**Which evaluators should I generate?** Treat the proposal as a candidate set — the suite below is intentionally broad so you can pick what matters for your team's quality bar. Reply with **which to keep, which to drop, and which to rename**; not every domain-specific proposal will fit your priorities. In `sdk_code` mode you may also add custom evaluators or change provider/model. In `publish` mode you may override `integration_provider`, `model_name`, `sampling_percentage`, `eval_scope`, `root_spans_only`, or `filter` per evaluator.

Do NOT proceed to code generation until the user confirms.

---

output_mode

• sdk_code

  → Phase 3A below

• data_only

  → skip to Phase 3B

• publish

  → skip to Phase 3C

1. Ground prompts in traces: LLMJudge system prompts and user prompts must reference patterns actually observed in production traces. Never write generic prompts like "evaluate whether the response is good" — ground them in the app's domain, observed failure patterns, and success criteria.
2. Keep template variables generic, add comments for context: Use

   {{input_data}}

   and

   {{output_data}}

   as top-level placeholders in prompts — do NOT reference nested span paths like

   {{input_data.messages[-1].content}}

   . The evaluator's data comes from the user's dataset and task function, not directly from spans. Instead, add a comment above each evaluator describing what data it was designed for and what the user should adapt:
   python

   # Designed for: input_data = user query, output_data = assistant response text
   # Observed from: root agent span (input.value → output.value)
   # If your dataset uses a different structure, adapt the prompt references below.

3. Use the narrowest evaluator type: If a check can be done with

   JSONEvaluator

   ,

   RegexMatchEvaluator

   ,

   StringCheckEvaluator

   , or

   LengthEvaluator

   , do NOT use an LLMJudge. Code-based evaluators are faster, cheaper, and deterministic.
4. BaseEvaluator subclasses:
   • Call

     super().__init__(name=name)

     in

     __init__

   • Return

     EvaluatorResult

     from

     evaluate()

   • Do NOT modify instance attributes in

     evaluate()

     (thread safety)
5. Names: Must match

   ^[a-zA-Z0-9_-]+$

   . Use snake_case descriptive names.
6. Imports: Consolidate at the top of the file. Only import classes that are actually used.
7. Evaluator list: Collect all evaluators into an

   evaluators

   list at the bottom of the file.
8. Anonymize PII: Strip emails, names, and sensitive data from any trace content included in LLMJudge prompts or the header comment.

1. Review: Check the generated prompts and criteria match your expectations
2. Test offline: Use

   LLMObs.experiment(evaluators=evaluators)

   to batch-evaluate against a labeled dataset and verify scores

undefined

• If

  rca_notebook_url

  was detected in Phase 0:

  An RCA notebook was created earlier in this session:

  {rca_notebook_url}

  Would you like to (a) append the evaluator suite summary to that notebook, or (b) create a new standalone notebook?

  If append: use the notebook creation fallback pattern (see below) with

  mcp__datadog-mcp__edit_datadog_notebook

  (

  id={rca_notebook_id}

  ,

  append_only=true

  , evaluator suite summary cell).
  If new: use the notebook creation fallback pattern (see below) with

  mcp__datadog-mcp__create_datadog_notebook

  .
• If no

  rca_notebook_url

  :

  Would you like to export this evaluator suite summary to a Datadog notebook?

  If yes: use the notebook creation fallback pattern (see below) with

  mcp__datadog-mcp__create_datadog_notebook

  :

  • name

    :

    Eval Bootstrap: {ml_app} — YYYY-MM-DD

  • type

    :

    report

  • cells

    : single markdown cell with the evaluator suite summary

  • time

    :

    { "live_span": "1h" }

create_datadog_notebook

edit_datadog_notebook

1. Try the MCP tool first.
2. If the MCP call fails, inspect the error:
   • Auth / permission error (401, 403) → stop and tell the user.
   • Field validation error (error names a specific field) → fix that field and retry the MCP call once.
   • Any other error (binding, serialization, unexpected response) → fall back to pup:
     • Write the payload to

       /tmp/nb_bootstrap_{ml_app}.json

       as a full API envelope:

       {"data": {"attributes": {"name": "...", "time": {...}, "cells": [...]}, "type": "notebooks"}}

     • Run

       pup notebooks create --file /tmp/nb_bootstrap_{ml_app}.json

     • If pup is not available either, render the notebook content as markdown in chat.
3. After successful creation by either method, output the URL:

   Evaluator suite exported to notebook: <url>

undefined

1. Review generated prompts in

   {output_path}

2. Run against a labeled dataset to validate scores
3. Deploy to Datadog LLM Experiments

---

• evaluators[].type

  :

  "llm_judge"

  for semantic evaluators;

  "code_check"

  for deterministic checks (regex, length, JSON validity, etc.).

• evaluators[].rubric

  : For

  llm_judge

  — full prompt text grounded in observed trace patterns. Use

  {{input}}

  and

  {{output}}

  as generic placeholders (not

  {{input_data}}

  — that's ddeval-specific). For

  code_check

  — null.

• evaluators[].implementation_hints.notes

  : Optional framework-agnostic guidance, e.g. "For OpenAI Evals, use

  rubric

  as a model-graded criterion. For Braintrust, use as an LLM scorer. For Promptfoo, use as an

  llm-rubric

  assertion."

• sample_records

  : 10–20 representative traces from Phase 1.

  suggested_labels

  are Claude's best-read from trace inspection — not ground truth. The field name communicates this explicitly.
• PII rule: Strip emails, names, and sensitive data from all

  input

  ,

  output

  , and

  evidence[].observation

  fields before writing (same as Phase 3A).