Autoresearch Agent

What This Skill Does

The Autoresearch Agent autonomously runs experiments to optimize a target file for a specific metric. It edits the file, evaluates the results, keeps improvements, discards failures, and loops indefinitely, providing a hands-off approach to optimization.

When to Use

• Making code faster, smaller, or better.
• Optimizing a file for a specific metric.
• Improving headlines, copy, or prompts.
• Running experiments overnight.
• Getting a specific metric from X to Y.

Key Features

• Sets up experiments with target file and evaluation metric.
• Automatically edits files and runs evaluations.
• Keeps improvements and discards failures.
• Supports project-level and user-level experiments.

Autoresearch Agent

You sleep. The agent experiments. You wake up to results.

Autonomous experiment loop inspired by Karpathy's autoresearch. The agent edits one file, runs a fixed evaluation, keeps improvements, discards failures, and loops indefinitely.

Not one guess — fifty measured attempts, compounding.

Slash Commands

When This Skill Activates

Recognize these patterns from the user:

"Make this faster / smaller / better" "Optimize [file] for [metric]" "Improve my [headlines / copy / prompts]" "Run experiments overnight" "I want to get [metric] from X to Y" Any request involving: optimize, benchmark, improve, experiment loop, autoresearch

If the user describes a target file + a way to measure success → this skill applies.

Setup

First Time — Create the Experiment

Run the setup script. The user decides where experiments live:

Project-level (inside repo, git-tracked, shareable with team): python scripts/setup_experiment.py
--domain engineering
--name api-speed
--target src/api/search.py
--eval "pytest bench.py --tb=no -q"
--metric p50_ms
--direction lower
--scope project

User-level (personal, in ~/.autoresearch/): python scripts/setup_experiment.py
--domain marketing
--name medium-ctr
--target content/titles.md
--eval "python evaluate.py"
--metric ctr_score
--direction higher
--evaluator llm_judge_content
--scope user

The --scope flag determines where .autoresearch/ lives: project (default) → .autoresearch/ in the repo root. Experiment definitions are git-tracked. Results are gitignored. user → ~/.autoresearch/ in the home directory. Everything is personal.

What Setup Creates

.autoresearch/ ├── config.yaml ← Global settings ├── .gitignore ← Ignores results.tsv, *.log └── {domain}/{experiment-name}/ ├── program.md ← Objectives, constraints, strategy ├── config.cfg ← Target, eval cmd, metric, direction ├── results.tsv ← Experiment log (gitignored) └── evaluate.py ← Evaluation script (if --evaluator used)

results.tsv columns: commit | metric | status | description commit — short git hash metric — float value or "N/A" for crashes status — keep | discard | crash description — what changed or why it crashed

Domains

If program.md Already Exists

The user may have written their own program.md. If found in the experiment directory, read it. It overrides the template. Only ask for what's missing.

Agent Protocol

You are the loop. The scripts handle setup and evaluation — you handle the creative work.

Before Starting

Read .autoresearch/{domain}/{name}/config.cfg to get:

• target — the file you edit
• evaluate_cmd — the command that measures your changes
• metric — the metric name to look for in eval output
• metric_direction — "lower" or "higher" is better
• time_budget_minutes — max time per evaluation Read program.md for strategy, constraints, and what you can/cannot change Read results.tsv for experiment history (columns: commit, metric, status, description) Checkout the experiment branch: git checkout autoresearch/{domain}/{name}

Each Iteration

Review results.tsv — what worked? What failed? What hasn't been tried? Decide ONE change to the target file. One variable per experiment. Edit the target file Commit: git add {target} && git commit -m "experiment: {description}" Evaluate: python scripts/run_experiment.py --experiment {domain}/{name} --single Read the output — it prints KEEP, DISCARD, or CRASH with the metric value Go to step 1

What the Script Handles (you don't)

Running the eval command with timeout Parsing the metric from eval output Comparing to previous best Reverting the commit on failure (git reset --hard HEAD~1) Logging the result to results.tsv

Starting an Experiment

Single iteration (the agent calls this repeatedly)

python scripts/run_experiment.py --experiment engineering/api-speed --single

Dry run (test setup before starting) python scripts/run_experiment.py --experiment engineering/api-speed --dry-run

Strategy Escalation

Runs 1-5: Low-hanging fruit (obvious improvements, simple optimizations) Runs 6-15: Systematic exploration (vary one parameter at a time) Runs 16-30: Structural changes (algorithm swaps, architecture shifts) Runs 30+: Radical experiments (completely different approaches) If no improvement in 20+ runs: update program.md Strategy section

Self-Improvement

After every 10 experiments, review results.tsv for patterns. Update the Strategy section of program.md with what you learned (e.g., "caching changes consistently improve by 5-10%", "refactoring attempts never improve the metric"). Future iterations benefit from this accumulated knowledge.

Stopping

Run until interrupted by the user, context limit reached, or goal in program.md is met Before stopping: ensure results.tsv is up to date On context limit: the next session can resume — results.tsv and git log persist

Rules

One change per experiment. Don't change 5 things at once. You won't know what worked. Simplicity criterion. A small improvement that adds ugly complexity is not worth it. Equal performance with simpler code is a win. Removing code that gets same results is the best outcome. Never modify the evaluator. evaluate.py is the ground truth. Modifying it invalidates all comparisons. Hard stop if you catch yourself doing this. Timeout. If a run exceeds 2.5× the time budget, kill it and treat as crash. Crash handling. If it's a typo or missing import, fix and re-run. If the idea is fundamentally broken, revert, log "crash", move on. 5 consecutive crashes → pause and alert. No new dependencies. Only use what's already available in the project.

Evaluators

Ready-to-use evaluation scripts. Copied into the experiment directory during setup with --evaluator.

Free Evaluators (no API cost)

LLM Judge Evaluators (uses your subscription)

| Evaluato