You sleep. The agent experiments. You wake up to results.

• "Make this faster / smaller / better"
• "Optimize [file] for [metric]"
• "Improve my [headlines / copy / prompts]"
• "Run experiments overnight"
• "I want to get [metric] from X to Y"
• Any request involving: optimize, benchmark, improve, experiment loop, autoresearch

python scripts/setup_experiment.py \
  --domain engineering \
  --name api-speed \
  --target src/api/search.py \
  --eval "pytest bench.py --tb=no -q" \
  --metric p50_ms \
  --direction lower \
  --scope project

~/.autoresearch/

python scripts/setup_experiment.py \
  --domain marketing \
  --name medium-ctr \
  --target content/titles.md \
  --eval "python evaluate.py" \
  --metric ctr_score \
  --direction higher \
  --evaluator llm_judge_content \
  --scope user

--scope

.autoresearch/

• project

  (default) →

  .autoresearch/

  in the repo root. Experiment definitions are git-tracked. Results are gitignored.

• user

  →

  ~/.autoresearch/

  in the home directory. Everything is personal.

.autoresearch/
├── config.yaml                        ← Global settings
├── .gitignore                         ← Ignores results.tsv, *.log
└── {domain}/{experiment-name}/
    ├── program.md                     ← Objectives, constraints, strategy
    ├── config.cfg                     ← Target, eval cmd, metric, direction
    ├── results.tsv                    ← Experiment log (gitignored)
    └── evaluate.py                    ← Evaluation script (if --evaluator used)

commit | metric | status | description

• commit

  — short git hash

• metric

  — float value or "N/A" for crashes

• status

  — keep | discard | crash

• description

  — what changed or why it crashed

1. Read

   .autoresearch/{domain}/{name}/config.cfg

   to get:

   • target

     — the file you edit

   • evaluate_cmd

     — the command that measures your changes

   • metric

     — the metric name to look for in eval output

   • metric_direction

     — "lower" or "higher" is better

   • time_budget_minutes

     — max time per evaluation
2. Read

   program.md

   for strategy, constraints, and what you can/cannot change
3. Read

   results.tsv

   for experiment history (columns: commit, metric, status, description)
4. Checkout the experiment branch:

   git checkout autoresearch/{domain}/{name}

1. Review results.tsv — what worked? What failed? What hasn't been tried?
2. Decide ONE change to the target file. One variable per experiment.
3. Edit the target file
4. Commit:

   git add {target} && git commit -m "experiment: {description}"

5. Evaluate:

   python scripts/run_experiment.py --experiment {domain}/{name} --single

6. Read the output — it prints KEEP, DISCARD, or CRASH with the metric value
7. Go to step 1

• Running the eval command with timeout
• Parsing the metric from eval output
• Comparing to previous best
• Reverting the commit on failure (

  git reset --hard HEAD~1

  )
• Logging the result to results.tsv

• Runs 1-5: Low-hanging fruit (obvious improvements, simple optimizations)
• Runs 6-15: Systematic exploration (vary one parameter at a time)
• Runs 16-30: Structural changes (algorithm swaps, architecture shifts)
• Runs 30+: Radical experiments (completely different approaches)
• If no improvement in 20+ runs: update program.md Strategy section

• Run until interrupted by the user, context limit reached, or goal in program.md is met
• Before stopping: ensure results.tsv is up to date
• On context limit: the next session can resume — results.tsv and git log persist

• One change per experiment. Don't change 5 things at once. You won't know what worked.
• Simplicity criterion. A small improvement that adds ugly complexity is not worth it. Equal performance with simpler code is a win. Removing code that gets same results is the best outcome.
• Never modify the evaluator.

  evaluate.py

  is the ground truth. Modifying it invalidates all comparisons. Hard stop if you catch yourself doing this.
• Timeout. If a run exceeds 2.5× the time budget, kill it and treat as crash.
• Crash handling. If it's a typo or missing import, fix and re-run. If the idea is fundamentally broken, revert, log "crash", move on. 5 consecutive crashes → pause and alert.
• No new dependencies. Only use what's already available in the project.

llm_judge_content

ctr_score

llm_judge_prompt

quality_score

llm_judge_copy

engagement_score

evaluate.py

• Claude Code Max → unlimited Claude calls for evaluation
• Codex CLI (ChatGPT Pro) → unlimited Codex calls
• Gemini CLI (free tier) → free evaluation calls

• TSV — default, tab-separated (compatible with spreadsheets)
• CSV — comma-separated, with proper quoting
• Markdown — formatted table, readable in GitHub/docs

• No evaluation command works → Test it before starting the loop. Run once, verify output.
• Target file not in git →

  git init && git add . && git commit -m 'initial'

  first.
• Metric direction unclear → Ask: is lower or higher better? Must know before starting.
• Time budget too short → If eval takes longer than budget, every run crashes.
• Agent modifying evaluate.py → Hard stop. This invalidates all comparisons.
• 5 consecutive crashes → Pause the loop. Alert the user. Don't keep burning cycles.
• No improvement in 20+ runs → Suggest changing strategy in program.md or trying a different approach.

• self-improving-agent — improves an agent's own memory/rules over time. NOT for structured experiment loops.
• senior-ml-engineer — ML architecture decisions. Complementary — use for initial design, then autoresearch for optimization.
• tdd-guide — test-driven development. Complementary — tests can be the evaluation function.
• skill-security-auditor — audit skills before publishing. NOT for optimization loops.