Autoresearch

Setup

autoresearch.md

autoresearch.md

autoresearch.jsonl

git log

1. Gather context: Ask (or infer from

   $ARGUMENTS

   and conversation) the Goal, Command to benchmark, Primary metric (name + direction), Files in scope, and Constraints.
2. Create branch:

   git checkout -b autoresearch/<goal>-<date>

   (e.g.

   autoresearch/test-speed-2026-03-21

   ).
3. Read source files: Understand the workload deeply before writing anything. Read every file in scope.
4. Write session files: Create

   autoresearch.md

   and

   autoresearch.sh

   (see templates below). If constraints require correctness validation (tests must pass, types must check), also create

   autoresearch.checks.sh

   . Commit all.
5. Run baseline: Execute the first experiment with no changes to establish the baseline metric.
6. Start looping: Begin the experiment loop immediately after the baseline is logged.

autoresearch.md

# Autoresearch: <goal>

## Objective
<Specific description of what we're optimizing and the workload.>

## Metrics
- **Primary**: <name> (<unit>, lower/higher is better)
- **Secondary**: <name>, <name>, ...

## How to Run
`./autoresearch.sh` — outputs `METRIC name=value` lines.

## Files in Scope
<Every file the agent may modify, with a brief note on what it does.>

## Off Limits
<What must NOT be touched — evaluation harness, data prep, etc.>

## Constraints
<Hard rules: tests must pass, no new deps, fixed time budget, etc.>

## What's Been Tried
<Update this section as experiments accumulate. Note key wins, dead ends,
and architectural insights so the agent doesn't repeat failed approaches.>

autoresearch.md

autoresearch.sh

#!/bin/bash
set -euo pipefail

# Pre-checks (fast, <1s — catch syntax errors early)
python3 -c "import ast; ast.parse(open('train.py').read())"

# Run benchmark
uv run train.py > /tmp/autoresearch-output.log 2>&1

# Extract and output metrics as METRIC lines
val_bpb=$(grep "^val_bpb:" /tmp/autoresearch-output.log | awk '{print $2}')
echo "METRIC val_bpb=$val_bpb"

• Use

  set -euo pipefail

  .
• Output

  METRIC name=value

  lines to stdout (one per metric). The primary metric name must match what's documented in

  autoresearch.md

  .
• Metric names: word chars, dots, or

  µ

  (e.g.

  val_bpb

  ,

  total_µs

  ,

  bundle.size_kb

  ).
• Keep the script fast — every second is multiplied by hundreds of runs.
• For fast/noisy benchmarks (<5s), run multiple times inside the script and report the median.
• Update the script during the loop as needed.

autoresearch.checks.sh

(optional)

#!/bin/bash
set -euo pipefail
pnpm test --run --reporter=dot 2>&1 | tail -50
pnpm typecheck 2>&1 | grep -i error || true

• Run it after every passing benchmark (exit 0).
• If checks fail, log the experiment as

  checks_failed

  and revert.
• Check execution time does NOT affect the primary metric.
• Keep output minimal — suppress verbose progress, only show errors.

The Experiment Loop

1. Formulate hypothesis: Based on prior results, source code understanding, and any ideas in

   autoresearch.ideas.md

   , choose what to try next.
2. Edit code: Modify the in-scope files. Make a single, focused change per experiment.
3. Commit:

   git add -A && git commit -m "<short description of what this experiment tries>"

4. Run benchmark:
   bash

   timeout 600 ./autoresearch.sh > run.log 2>&1

   If the command times out or crashes, treat it as a failure.
5. Parse metrics: Extract

   METRIC

   lines from the output:
   bash

   grep '^METRIC ' run.log

   If no METRIC lines found, the run crashed — read

   tail -50 run.log

   for the error.
6. Run checks (if

   autoresearch.checks.sh

   exists and benchmark passed):
   bash

   timeout 300 ./autoresearch.checks.sh > checks.log 2>&1

7. Evaluate and log:
   • Improved (primary metric better than best so far) → status

     keep

     . The commit stays.
   • Worse or equal → status

     discard

     . Revert: stage autoresearch files first, then reset.
   • Crash (benchmark failed) → status

     crash

     . Fix if trivial, otherwise revert and move on.
   • Checks failed → status

     checks_failed

     . Revert.
8. Log to JSONL: Append one line to

   autoresearch.jsonl

   :
   json

   {"run":1,"commit":"a1b2c3d","metric":0.9979,"metrics":{"val_bpb":0.9979,"peak_vram_mb":45060.2},"status":"keep","description":"baseline","timestamp":1711036800000,"confidence":null}

9. On discard/crash/checks_failed — revert code changes:
   bash

   # Preserve autoresearch session files, revert everything else
   git add autoresearch.jsonl autoresearch.md autoresearch.sh autoresearch.ideas.md autoresearch.checks.sh 2>/dev/null || true
   git checkout -- .
   git clean -fd

10. Check confidence: After 3+ runs, run the confidence script from the skill's installation directory:
    bash

    bash "$(dirname "$(readlink -f "$0")")/scripts/confidence.sh"

    Or locate it via the skill path and run it directly. Interpret the score:
    • >= 2.0x: Improvement is likely real (green).
    • 1.0-2.0x: Above noise but marginal (yellow).
    • < 1.0x: Within noise — consider re-running to confirm (red).
11. Update session: Periodically update

    autoresearch.md

    "What's Been Tried" section and run the summary script to review progress.

JSONL Schema

autoresearch.jsonl

run

commit

metric

metrics

status

keep

discard

crash

checks_failed

description

timestamp

confidence

Resuming

autoresearch.md

1. Read

   autoresearch.md

   for full context (objective, what's been tried, constraints).
2. Read

   autoresearch.jsonl

   to reconstruct state (best metric, run count, last segment).
3. Read

   git log --oneline -20

   for recent commit history.
4. Check

   autoresearch.ideas.md

   if it exists — prune stale entries, experiment with promising ones.
5. Continue the loop from where it left off. Do not re-run the baseline.

Ideas Backlog

autoresearch.ideas.md

autoresearch.md

Loop Rules

references/loop-rules.md

• Primary metric is king. Improved → keep. Worse/equal → discard.
• Simpler is better. Remove code for equal perf = keep. Ugly complexity for tiny gain = discard.
• Don't thrash. Repeatedly reverting the same idea? Try something structurally different.
• Think longer when stuck. Re-read source files, reason about what the CPU/compiler/runtime is actually doing. Deep understanding beats random variation.
• Crashes: fix if trivial (typo, missing import), otherwise log and move on. Don't over-invest.
• NEVER STOP. The user may be away for hours. Keep going until interrupted.

User Messages During Experiments