ARA Seal Level 2: Semantic Epistemic Review

level2_report.json

Six Review Dimensions

Procedure

Step 1: Read the ARA

read_order

1. PAPER.md

2. logic/claims.md

3. logic/experiments.md

4. logic/problem.md

5. logic/concepts.md

6. logic/solution/architecture.md

   ,

   algorithm.md

   ,

   constraints.md

   ,

   heuristics.md

7. logic/related_work.md

8. trace/exploration_tree.yaml

9. evidence/README.md

   (if exists)
10. Spot-check 2-3 evidence files from

    evidence/tables/

    or

    evidence/figures/

Step 2: Parse Entities

logic/claims.md

## C{NN}: {title}

• Statement

  ,

  Status

  ,

  Falsification criteria

  ,

  Proof

  (experiment IDs),

  Dependencies

  (claim IDs),

  Tags

logic/experiments.md

## E{NN}: {title}

• Verifies

  (claim IDs),

  Setup

  ,

  Procedure

  ,

  Metrics

  ,

  Expected outcome

  ,

  Baselines

  ,

  Dependencies

logic/solution/heuristics.md

## H{NN}

• Rationale

  ,

  Sensitivity

  ,

  Bounds

  ,

  Code ref

logic/problem.md

O{N}

G{N}

trace/exploration_tree.yaml

id

type

title

failure_mode

lesson

choice

alternatives

result

Step 3: Build Working Maps

• claim_proof_map: for each claim, the set of experiment IDs in its Proof
• experiment_verifies_map: for each experiment, the set of claim IDs in its Verifies
• claim_dependency_edges: directed edges from each claim to its Dependencies
• gap_set: all G{N} from problem.md
• rejected_nodes: exploration tree nodes with type =

  dead_end

  or

  pivot

• decision_nodes: exploration tree nodes with type =

  decision

Step 4: Evaluate Each Dimension

D1. Evidence Relevance

• Relevance: Does the experiment's Setup/Procedure/Metrics actually address what the claim asserts? (Not just "link exists" but "link is substantively relevant.")
• Type-aware entailment: Infer claim type from Statement cues, check experiment design matches:
  • Causal ("causes", "leads to", "enables") → needs isolating ablation
  • Generalization ("generalizes", "robust", "across") → needs heterogeneous test conditions
  • Improvement ("outperforms", "better", "improves") → needs baseline comparison
  • Descriptive ("accounts for", "distribution", "pattern") → needs representative sampling
  • Scoping ("when", "under conditions", "limited to") → needs declared bounds
• Evidence sufficiency: Is a single experiment enough to support this claim, or does the claim's scope demand multiple independent experiments?

• 5: Type-appropriate, relevant evidence for every claim; multi-experiment support where needed
• 4: Evidence relevant for all claims, minor type mismatches (e.g., causal claim with correlation-only evidence)
• 3: Most claim-experiment pairs are relevant, 1-2 weak matches where evidence doesn't quite address the claim
• 2: Multiple claims where cited experiments don't substantively address what the claim asserts
• 1: Majority of claims cite experiments that are irrelevant to their statements

D2. Falsifiability Quality

• Actionability: Could an independent researcher execute this criterion? Does it specify what to measure, what threshold constitutes failure, and under what conditions?
• Non-triviality: Is the criterion non-tautological? ("If the method doesn't work" is trivial. "Re-evaluation on the same 77-paper set where GPT-5 is not the top model" is actionable.)
• Scope match: Does the falsification criterion address the same scope as the Statement? (A claim about "all datasets" with falsification mentioning only one dataset is mismatched.)
• Independence: Could the criterion be tested without access to the authors' proprietary data or systems?

• 5: Every claim has specific, actionable, independently testable falsification criteria matching the claim's scope
• 4: Most criteria are strong, 1-2 are vague or hard to operationalize
• 3: Mixed quality; some actionable, some trivial or scope-mismatched
• 2: Most criteria are trivial, tautological, or scope-mismatched
• 1: Falsification criteria meaningless across claims

D3. Scope Calibration

• Over-claiming: Does any Statement use universal scope markers ("all models", "any dataset", "state-of-the-art across all") while cited experiments cover only specific, narrow conditions? The gap must be substantial.
• Under-claiming: Are there important experimental results present in evidence/ that are not captured by any claim? (Evidence without a corresponding claim.)
• Assumption explicitness: Are key assumptions stated in problem.md (Assumptions section) or constraints.md? Are there unstated assumptions implied by the experimental design?
• Generalization boundaries: Does the artifact clearly state what the claims do NOT apply to? Check constraints.md and limitations in the exploration tree.
• Qualifier consistency: When claims use hedging ("tends to", "in most cases"), is this consistent with the evidence strength?

• 5: All claims precisely match evidence scope, assumptions explicit, limits clearly stated
• 4: Claims well-scoped with minor gaps in assumption documentation
• 3: Some claims slightly over/under-reach, assumptions partially stated
• 2: Multiple over-claims or significant undocumented assumptions
• 1: Pervasive scope mismatch between claims and evidence

D4. Argument Coherence

• Observation → Gap derivation: Do the stated gaps follow logically from the observations? Or are they asserted without connection?
• Gap → Insight connection: Does the key insight in problem.md address the identified gaps?
• Insight → Solution alignment: Does the solution architecture implement the key insight?
• Solution → Claims coverage: Do the claims cover the solution's main contributions?
• Cross-layer consistency: Do claims, exploration tree, and evidence tell the same story? Flag contradictions.
• Narrative completeness: Are there motivating questions from problem.md that are neither answered nor explicitly deferred?
• Gap coverage: For each gap in problem.md, is there at least one claim that substantively addresses it? Flag gaps that are motivated but never resolved.

• 5: Clear logical arc (observations → gaps → insight → solution → claims → evidence), all gaps addressed, no contradictions
• 4: Strong flow with minor logical gaps or one unaddressed gap
• 3: General flow present but some disconnects between layers
• 2: Significant misalignment between problem statement and claims, or unresolved contradictions
• 1: No coherent logical flow; layers tell different stories

D5. Exploration Integrity

• Dead-end quality: Is the

  failure_mode

  specific enough to be actionable? ("Didn't work" is bad. "Divergence after 1000 steps due to gradient explosion" is good.) Is the

  lesson

  a genuine transferable insight?
• Decision rationale quality: Do rationales explain WHY the chosen path was preferred over alternatives? Are alternatives real alternatives or strawmen?
• Rebutted-branch consistency: Does any claim advocate an approach marked as dead_end or pivot in the tree? (This is a logical contradiction.)
• Exploration breadth: For the paper's main design choices, were at least 2 alternatives considered and documented?
• Honesty signal: Does the tree document genuine negative results, or does it read like a post-hoc justification? A tree with zero dead-ends or only trivial failures is suspicious.

• 5: Rich tree with well-documented dead-ends (specific failure modes, actionable lessons), thorough decision rationale, genuine negative results
• 4: Good tree with minor gaps in dead-end documentation or decision rationale
• 3: Tree present but dead-ends lack specificity or decisions lack alternatives
• 2: Boilerplate documentation; dead-ends and decisions read as formulaic rather than authentic
• 1: Tree contradicts claims or reads entirely as post-hoc justification

D6. Methodological Rigor

• Baseline adequacy: Are the right things being compared? Are baselines recent and relevant? Flag experiments with "no baseline" for comparative claims.
• Ablation coverage: For claims involving multiple components, does at least one experiment isolate individual contributions?
• Statistical reporting: Do experiments mention variance, confidence intervals, number of runs, or statistical tests? Flag single-run results for quantitative claims.
• Metric-claim alignment: Does the metric actually measure what the claim asserts? (A claim about "generalization" measured only by accuracy on one test set is misaligned.)
• Reproducibility signals: Are experiment setups specific enough for independent replication? (Model name, dataset, hardware, hyperparameters.)

• 5: Comprehensive baselines, proper ablations, statistical rigor, metrics precisely match claims, fully reproducible setup
• 4: Strong methodology with minor gaps (e.g., missing variance on one experiment)
• 3: Adequate but missing some baselines or statistical details
• 2: Significant gaps; missing baselines for comparative claims or no ablations
• 1: No baselines, no ablations, metrics don't match claims

Step 5: Compile Findings

• finding_id: F01, F02, ... (sequential)
• dimension: which of D1-D6
• severity: one of:

  • critical

    — fundamental epistemic flaw; the claim or argument cannot stand as written

  • major

    — significant weakness that undermines a claim or dimension score

  • minor

    — noticeable issue that doesn't invalidate the work

  • suggestion

    — constructive improvement opportunity, not a flaw
• target_file: which ARA file
• target_entity: C{NN}, E{NN}, H{NN}, G{N}, or node ID (if applicable)
• evidence_span: verbatim substring from the ARA that triggered the finding (MUST be exact quote; omit if the finding is about an absence)
• observation: what you found (factual)
• reasoning: why it matters (analytical)
• suggestion: how to fix or improve it (constructive)

Step 6: Compute Overall Grade

Step 7: Write Report

level2_report.json

{
  "artifact": "<name>",
  "artifact_dir": "<path>",
  "review_version": "3.0.0",
  "prerequisite": "Level 1 passed",

  "overall": {
    "grade": "Accept",
    "mean_score": 4.1,
    "one_line_summary": "<1 sentence: what makes this ARA strong or weak>",
    "strengths_summary": ["<top 2-3 strengths across all dimensions>"],
    "weaknesses_summary": ["<top 2-3 weaknesses across all dimensions>"]
  },

  "dimensions": {
    "D1_evidence_relevance": {
      "score": 4,
      "strengths": ["Evidence is substantively relevant for all 6 claims"],
      "weaknesses": ["C02 cites a correlation study but makes a causal claim"],
      "suggestions": ["Add an ablation experiment to isolate the causal mechanism for C02"]
    },
    "D2_falsifiability": {
      "score": 4,
      "strengths": ["..."],
      "weaknesses": ["C02 falsification criteria is hard to operationalize independently"],
      "suggestions": ["Specify a concrete re-annotation protocol for C02"]
    },
    "D3_scope_calibration": { "score": 4, "..." : "..." },
    "D4_argument_coherence": { "score": 4, "..." : "..." },
    "D5_exploration_integrity": { "score": 3, "..." : "..." },
    "D6_methodological_rigor": { "score": 4, "..." : "..." }
  },

  "findings": [
    {
      "finding_id": "F01",
      "dimension": "D6_methodological_rigor",
      "severity": "major",
      "target_file": "logic/experiments.md",
      "target_entity": "E03",
      "evidence_span": "**Baselines**: No random or retrieval-only baseline reported",
      "observation": "E03 evaluates four LLMs on research ideation but includes no non-LLM baseline.",
      "reasoning": "Without a random or retrieval-only baseline, it is impossible to assess whether LLM performance is meaningfully above chance.",
      "suggestion": "Add a retrieval-only baseline (e.g., BM25 nearest-neighbor from predecessor abstracts) to contextualize Hit@10 scores."
    }
  ],

  "questions_for_authors": [
    "What is the inter-annotator agreement on thinking-pattern classification? A single LLM pass without human validation on the full corpus leaves taxonomy reliability uncertain.",
    "..."
  ],

  "read_order": ["PAPER.md", "logic/claims.md", "..."]
}

Critical Rules

1. Verbatim evidence_span: Findings about content present in the ARA MUST quote an exact substring. Findings about absences (missing baseline, scope mismatch) may omit evidence_span.
2. Constructive tone: Every weakness must come with a suggestion. You are helping authors improve, not punishing them.
3. Calibrated scoring: Most competent ARAs should land in the 3-4 range. A score of 5 means genuinely excellent, not just "no problems found." A score of 1 means fundamental problems, not just "could be better."
4. No false grounding: Support must flow through Proof → experiments.md → evidence/. Agreement in prose (problem.md, architecture.md) does not substitute for experimental evidence.
5. Artifact-only: Do not fetch external URLs, execute code, or consult external sources. Take the ARA's reported evidence at face value.
6. Balanced review: Actively look for strengths, not just weaknesses. A review that only lists problems is not useful.
7. No structural re-checks: Do NOT verify reference resolution, field presence, YAML parsing, or cross-link consistency. Level 1 has already validated all of this. Focus entirely on whether the content is epistemically sound.

Reference