dashboard/serve.py

1. Complete the stage's analysis
2. Write stage data to a JSON file (e.g.,

   stage-0-data.json

   )
3. Launch:

   python dashboard/serve.py --stage <name> --data <file>.json

4. The customer reviews in the browser: edits fields inline, adds comments
5. Read the feedback JSON file after the customer clicks Confirm or Request Changes
6. If confirmed → generate final deliverables (docx, CSV) and proceed to next stage
7. If changes requested → apply feedback, regenerate, re-launch dashboard

xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

• If they provide a tenant ID: Use

  /clone-agent

  to connect to their Copilot Studio environment. Pull the agent's topics, knowledge sources, and configuration. Use this as the ground truth for Stage 0 (Discover) — pre-fill the Agent Vision from the actual agent config, then confirm with the customer.
• If they don't have a tenant ID or agent yet: Say: "No problem — we can work from a description instead. I'll walk you through defining what the agent should do, and we'll build the eval plan from that." Proceed with the description-based flow below.

This is the default and preferred path. Working from a connected agent produces more accurate eval plans because you can see the actual topics, triggers, knowledge sources, and boundaries rather than relying on the customer's verbal description.

1. What problem does the agent solve?
   • "Tell me about the agent you're building (or planning to build). What's the core problem it solves for your users?"
2. Who are the users?
   • "Who will talk to this agent? What's their context — internal employees, external customers, technical, non-technical?"
3. What will the agent know?
   • "What information sources will the agent use? Policy docs, FAQs, databases, APIs?"
   • If they're not sure: "That's fine — we'll plan around what you expect to have."
4. What should the agent DO vs NOT DO?
   • "What are the boundaries? What should the agent never attempt to answer or do?"
5. What does success look like?
   • "If the agent is working perfectly, what does that look like? How would you know?"
6. What happens if the agent gets it wrong?
   • "What's the worst case if it gives a bad answer? Is this internal low-risk, or customer-facing high-risk?"
7. Does the agent behave differently per user?
   • "Does the agent return different results depending on who's asking? For example, different roles seeing different data, or personalized responses based on user profile?"
   • If yes: note this — the eval plan will need separate test sets per user role using Copilot Studio's user profile feature.

1. Write the Agent Vision to

   stage-0-data.json

   :
   json

   {"agent_name": "...", "vision": {"purpose": "...", "users": "...", "knowledge": [...], "capabilities": [...], "boundaries": [...], "success_criteria": "...", "role_based_access": false, "risk_profile": "medium"}}

2. Launch the dashboard:
   bash

   python dashboard/serve.py --stage discover --data stage-0-data.json

3. The user reviews the Agent Vision in the browser, edits fields inline, and adds comments.
4. When the user clicks Confirm & Continue, read

   discover-feedback.json

   :
   • If

     status

     is

     "confirmed"

     : Apply any edits from the

     edits

     field, then proceed to Stage 1.
   • If

     status

     is

     "changes_requested"

     : Apply the feedback, regenerate the Agent Vision, and re-launch the dashboard.
5. Only proceed to Stage 1 after the user confirms.

1. Determine eval depth from agent architecture:
   Different agent architectures require different eval layers. Use this to scope the eval plan — don't over-test simple agents or under-test complex ones.

   Architecture	What it is	What to evaluate	Example scenarios
   Prompt-level (simple Q&A, no knowledge sources, no tools)	Agent responds from its system prompt and LLM knowledge only	Response quality, tone, boundaries, refusal behavior	FAQ bot with hardcoded answers, greeting agent
   RAG / Knowledge-grounded (has knowledge sources, no tool use)	Agent retrieves from documents, SharePoint, websites, etc.	Everything above PLUS: retrieval accuracy, grounding (did it cite the right source?), hallucination prevention, completeness	HR policy bot, IT knowledge base agent
   Agentic (multi-step, tool use, orchestration)	Agent calls APIs, uses connectors, makes decisions, chains actions	Everything above PLUS: tool selection accuracy, action correctness, error recovery, multi-turn context retention, task completion rate	Expense submission agent, incident triage bot, booking agent

   Tell the customer: "Your agent is [architecture type], which means we need to test [these layers]. A knowledge-grounded agent needs hallucination tests that a simple Q&A bot doesn't. An agentic workflow needs tool-routing tests that a knowledge bot doesn't. This scopes your eval so you're testing what actually matters."
   Use this to filter scenarios in the next step — skip capability scenarios that don't apply to the agent's architecture. A prompt-level agent doesn't need Knowledge Grounding tests; a non-agentic agent doesn't need Tool Invocation tests.
2. Match to scenario types:

1. Produce the eval plan:

1. Write the plan to

   stage-1-data.json

   using the Zone Model structure:
   json

   {
     "agent_name": "...",
     "scenarios": [
       {
         "id": 1,
         "name": "PTO policy lookup",
         "zone": "1",
         "quality_dimension": "Policy Accuracy",
         "acceptance_criteria": "Pass = agent returns correct PTO accrual rate matching HR policy.\nFail = incorrect rate or information from wrong tenure band.",
         "target_pass_rate": 90
       }
     ],
     "quality_dimensions": ["Policy Accuracy", "Source Attribution", "Hallucination Prevention", ...]
   }

   Each scenario must have:

   zone

   ("1", "2", or "3"),

   quality_dimension

   ,

   acceptance_criteria

   (with Pass = and Fail = conditions), and

   target_pass_rate

   (increment of 5%).
   Zone assignment guidelines:
   • Zone 1 ("This Must Work"): Safety failures, core business functions, guardrail tests. Highest value, full investment.
   • Zone 2 ("Real but Lower Priority"): Nice-to-have features, low-traffic use cases, areas where higher score doesn't justify effort.
   • Zone 3 ("Exploratory"): Emergent patterns, new tools with minimal effort, capabilities you're exploring.
2. Launch the dashboard:
   bash

   python dashboard/serve.py --stage plan --data stage-1-data.json

3. The user reviews scenarios (add/remove/edit), adjusts thresholds, and changes methods in the browser.
4. When the user confirms, read

   plan-feedback.json

   and apply edits. If changes requested, regenerate and re-launch.
5. After confirmation, automatically generate the customer-ready .docx eval plan report using the

   /docx

   skill — do not wait for the user to ask for it. This is the customer's first deliverable.

• Concise — no filler, no walls of text. Tables over paragraphs.
• Presentable — professional formatting with color-coded headers, clean tables, visual hierarchy
• Self-contained — a customer who wasn't in the conversation can read it and understand the eval plan

1. Agent Vision summary (from Stage 0) — 5-6 lines max
2. Zone Model overview — explain the three zones and how scenarios were classified based on Priority + Value + Effort
3. Zone assignment table — scenarios grouped by zone (Zone 1: "This Must Work", Zone 2: "Real but Lower Priority", Zone 3: "Exploratory") with acceptance criteria (Pass = / Fail = conditions)
4. Quality Dimensions to Test — list dimensions with grouped scenarios under each
5. Success Metrics table — Zone, Scenario, Quality Dimension, Target Pass Rate for each scenario
6. Method mapping explanation — which test methods apply to which quality dimensions

• The agent walks users through multi-step processes (e.g., troubleshooting, onboarding, form completion)
• Context retention matters — later answers depend on earlier ones
• The agent needs to ask clarifying questions before answering
• The scenario involves slot-filling or information gathering across turns

• Each question is independent (FAQ, policy lookup, data retrieval)
• You need Compare meaning, Text similarity, or Exact match (conversation mode doesn't support these)
• You need more than 20 test cases in a set

1. Generate one test case per scenario row from the plan. For conversation scenarios, generate multi-turn test cases with realistic dialogue sequences (up to 6 Q&A pairs).
2. Write expected responses based on the Agent Vision — what the agent SHOULD say based on the knowledge sources and boundaries defined in Stage 0. Note: "These expected responses reflect your stated requirements. Refine them once the agent is built and you see how it actually responds."
3. Group by quality signal into separate CSV files:

   • eval-knowledge-accuracy.csv

   • eval-safety-compliance.csv

   • eval-hallucination-prevention.csv

   • eval-routing.csv

   • eval-robustness.csv

   • eval-personalization.csv

     (if applicable)
   Only create files for categories that apply.
4. CSV format — Copilot Studio import format:

"Question","Expected response","Testing method"
"How many PTO days do LA employees get?","LA employees receive 18 PTO days per year.","Compare meaning"

General quality

Compare meaning

Similarity

Exact match

Keyword match

1. Test method per scenario type:

1. Highlight the value: "You now have [X] test cases across [Y] quality signals. Compare that to the 5-10 happy-path prompts most customers start with. These include adversarial attacks, hallucination traps, robustness tests, and edge cases your users will encounter in production."

1. Write the test cases to

   stage-2-data.json

   using the Zone Model structure:
   json

   {
     "agent_name": "...",
     "test_sets": [
       {
         "quality_dimension": "Policy Accuracy",
         "methods": ["Compare meaning", "Keyword match"],
         "scenarios": [
           {
             "scenario_id": 1,
             "name": "PTO policy lookup",
             "zone": "1",
             "acceptance_criteria": "Pass = correct PTO accrual rate.\nFail = incorrect rate.",
             "cases": [
               {"id": 1, "question": "...", "expected_response": "Text with [VERIFY: factual content to check] markers"}
             ]
           }
         ]
       }
     ]
   }

   Key requirements:
   • Group test cases by

     quality_dimension

     (not quality_signal), with

     scenarios

     nested under each dimension
   • Each scenario carries its

     zone

     ,

     acceptance_criteria

     (from Stage 1), and

     cases

   • Test

     methods

     are set at the dimension level, not per-case
   • Wrap AI-generated factual content in

     [VERIFY: ...]

     markers so the dashboard highlights them for human review
2. Launch the dashboard:
   bash

   python dashboard/serve.py --stage generate --data stage-2-data.json

3. The user reviews test cases per quality dimension tab (zone-colored), reviews acceptance criteria per scenario group, checks VERIFY-highlighted factual content, and edits expected responses inline.
4. When the user confirms, read

   generate-feedback.json

   and apply all edits. If changes requested, regenerate and re-launch.
5. After confirmation, generate the final deliverables:

"Question","Expected response","Testing method"

/docx

• Concise — no filler, no walls of text. Tables over paragraphs.
• Presentable — professional formatting with color-coded headers, clean tables, visual hierarchy
• Self-contained — a customer who wasn't in the conversation can read it and understand the eval plan + test cases

1. Agent Vision summary (from Stage 0) — 5-6 lines max
2. Zone Model summary — scenarios by zone with acceptance criteria (Pass/Fail)
3. Test cases organized by quality dimension, with scenario groups showing zone badge and acceptance criteria
4. For each test case: Question, Expected Response (with [VERIFY] content called out), and suggested test method
5. Summary table: quality dimension, scenario count, test case count, methods
6. "What these tests catch" callout — 3-4 bullet points on what the customer would have missed
7. Next steps — what to do with these files

eval-runner.js

node eval-runner.js --token-endpoint "<URL>" --csv-dir .

/chat-with-agent

• Compare meaning

  → semantic equivalence (0.0-1.0)

• General quality

  → helpfulness/accuracy/relevance (0.0-1.0)

• Keyword match

  → code-based string matching

• Exact match

  → code-based string equality

ANTHROPIC_API_KEY

1. Pre-triage check — Were knowledge sources accessible? APIs healthy? Auth valid?
2. Score summary — Total, passed, failed, pass rate per category and test method.
3. Failure triage — Explain the key insight: "Before we blame the agent — at least 20% of failures in a new eval are actually eval setup issues, not agent issues. The test case might be wrong, the expected response might be outdated, or the testing method might be inappropriate. Let me check that first."
   Apply 5-question eval verification for each failure.
4. Root causes: Eval Setup Issue / Agent Configuration Issue / Platform Limitation.
5. Top 3 actions — Each: Change X → Re-run Y → Expect Z.
6. Pattern analysis and next-run recommendation.

1. Write the triage data to

   stage-4-data.json

   using the success metrics structure:
   json

   {
     "agent_name": "...",
     "summary": {"total": 28, "passed": 19, "failed": 9},
     "success_metrics": [
       {"scenario_id": 1, "name": "PTO policy lookup", "zone": "1", "quality_dimension": "Policy Accuracy", "target_pass_rate": 90, "actual_pass_rate": 100, "cases_total": 2, "cases_passed": 2}
     ],
     "eval_results": [
       {"scenario_id": 1, "question": "...", "expected": "...", "actual": "...", "method": "Compare meaning", "score": 0.92, "pass": true, "explanation": "Rationale from LLM judge..."}
     ],
     "failures": [
       {"id": 1, "scenario_id": 2, "scenario_name": "...", "zone": "1", "quality_dimension": "...", "question": "...", "expected": "...", "actual": "...", "root_cause": "agent_config", "explanation": "..."}
     ],
     "top_actions": [...],
     "patterns": [...]
   }

   Key requirements:

   • success_metrics

     maps each scenario to its target vs actual pass rate (from Stage 1 targets)

   • eval_results

     contains ALL test case results (not just failures) so scenarios can be expanded in the dashboard
   • Each eval result includes

     explanation

     (the LLM judge rationale) for human review
   • No

     verdict

     field — the dashboard shows success metrics status per zone instead of SHIP/ITERATE/BLOCK
2. Launch the dashboard:
   bash

   python dashboard/serve.py --stage interpret --data stage-4-data.json

3. The user reviews success metrics status per zone, expands scenario rows to see test case details, uses Human Judgement (Agree/Disagree) to override LLM judge assessments, and re-classifies root causes.
4. When the user confirms, read

   interpret-feedback.json

   and apply edits (including human disagrees which become eval_setup root causes). If changes requested, regenerate and re-launch.
5. After confirmation, generate the customer-ready .docx triage report using the

   /docx

   skill. Same principles: concise, presentable, self-contained. Structure:
   1. Success Metrics Status — zone summary cards (targets met per zone) + full scenario table (zone, scenario, quality dimension, target vs actual, status)
   2. Failure triage table (zone, scenario, question, expected, actual, root cause) — include human-disagreed entries as "Eval Setup — Human Disagrees"
   3. Top actions (Change → Re-run → Expect)
   4. Pattern analysis — zone-aware patterns highlighting systemic issues
   5. Next steps

• CSV headers stay English (Copilot Studio requirement)
• Technical terms in English with Chinese parenthetical on first use: Compare meaning (语义比较), General quality (综合质量), Keyword match (关键词匹配), Exact match (精确匹配)

• No user profiles — they can't assign a test account to simulate authenticated users during evaluation
• No Text Similarity method — all other test methods work normally These are documented at About agent evaluation. Don't let them design an eval plan around features they can't use.

• Discover first — understand the agent's purpose and the customer's expectations before anything else.
• No running agent required for Stages 0-2. The skill works from a description, an idea, or a conversation.
• Explain your reasoning. Don't just output artifacts — narrate WHY you're making each choice. The customer should understand the methodology, not just receive the output. This is what makes them self-sufficient.
• Highlight what they'd miss. At each stage, point out the scenarios, methods, or insights the customer wouldn't have thought of on their own — hallucination tests, adversarial cases, the "20% are eval bugs" insight.
• Be specific — use real names, real scenarios. No generic advice.
• Always include at least 1 adversarial/safety scenario.
• Keep everything in the CLI unless asked otherwise.
• Pause between stages for confirmation.
• Match the user's language.