scorecard --live-check

• Substring-match results that coincidentally contain the query but don't match semantically (e.g., a query matches a substring of a larger unrelated term)
• Aggregation commands silently dropping sources when only some of the requested N come back
• Ranking or sort commands returning top-N results that aren't plausibly the best for the query (broken weights, extractor fallbacks)
• URLs in output pointing at category index pages, feed endpoints, or random-selector routes rather than canonical content permalinks
• Format bugs the rule-based layer doesn't catch (mojibake, inconsistent pluralization, truncated/wrapped cell content)

Review the sampled outputs from the shipped CLI at

$CLI_DIR

. You have these ground-truth sources:

• Sampled command output: read

  /tmp/output-review-livecheck.json

  and inspect the

  live_check.features[]

  array. Each entry has the command, example invocation, actual stdout (in

  output_sample

  , bounded to ~4 KiB), the pass/fail reason, and a

  warnings

  array (populated by rule-based checks like the raw-HTML-entity detector).
• Review only

  status: pass

  entries. Entries with

  status: fail

  either crashed, timed out, or had placeholder args (

  <id>

  ,

  <url>

  ) that never produced real output — their sample is empty and there's nothing for you to judge. Phase 5 dogfood handles test-coverage and exit-code concerns.

• $CLI_DIR/research.json

  novel_features

  (planned behavior per feature) and

  novel_features_built

  (verified built commands).
• The CLI binary at

  $CLI_DIR/<cli-name>-pp-cli

  — you may invoke additional commands to gather more output when a finding needs verification.

For each of these checks, report findings under 50 words each. Only report issues a human user would notice in 5 minutes of hands-on testing — not every edge case a thorough QA pass might find:

1. Output semantically matches query intent. For sampled novel features with a query argument, judge relevance beyond what the mechanical query-token check in live-check already enforced. A feature that passed live-check's

   outputMentionsQuery

   test still contains some query token somewhere — but "buttermilk" appearing as a substring of "butter" results, or "brownies" returning a chili recipe because the extractor fell back to adjacent content, both slip past the mechanical check. Only flag when a human user would look at the top results and say "this isn't what I asked for." Skip this check when the example has no query argument.
2. No obvious format bugs. Does the output contain raw HTML entities, mojibake (question marks or replacement chars in titles), or malformed URLs (pointing at category index pages, feed endpoints, or random-selector routes rather than canonical content permalinks)? Rule-based live-check catches numeric entities; this layer catches the broader class.
3. Aggregation commands show all requested sources. For commands with a

   --source

   /

   --site

   /

   --region

   CSV flag: if the user requested N sources, does output show N, or does stderr explain the missing ones? Silent drops of failed sources are a top failure mode for fan-out commands.
4. Result ordering/ranking makes sense. For commands that claim to rank or sort, does the top result look plausibly best given the query? Watch for broken score weights, off-by-one sort bugs, and silent fallback to recency when relevance computation fails.

Return a list of findings. For each: check name, severity (

warning

in Wave B;

error

reserved for Wave C), one-line description, one-sentence fix suggestion. If the CLI passes all four checks, return "PASS — no findings."

• All findings surface as

  warning

  — never

  error

  . Shipcheck proceeds regardless.
• The caller logs findings to the run's artifact directory (e.g.,

  manuscripts/<api>/<run>/proofs/phase-4.85-findings.md

  ) and surfaces them to the user. Findings are not persisted to

  scorecard.json

  — that path is reserved for Wave C.
• The user decides case by case whether to fix before shipping.

• If stdout is not a TTY, callers follow fail-open-with-log: findings recorded, shipcheck proceeds without prompting.

• status: SKIP

  (reviewer crash, timeout, missing data) is informational — shipcheck does not block on it.
• No

  --auto-approve-warnings

  flag yet. The policy is already "warnings don't block" in Wave B, so the flag has no effect to gate.

error

• Can't verify numeric accuracy (prices, ratings, rankings vs ground-truth). If the CLI says a recipe has 4.8 stars and it actually has 4.2, this skill won't catch it.
• Can't detect data-freshness issues (recipe published 2019 vs 2024). These need live comparison against authoritative sources.
• Can't judge subjective preferences ("is this the best recipe for chocolate chip cookies?").
• Sampled outputs only — covers the commands in

  live_check.features[]

  . Full command-tree coverage belongs in Phase 5 dogfood.
• Non-English output: the reviewer's query-intent check assumes English-language query/output. For non-English CLIs, calibrate the prompt separately.