Autoresearch

Set up and run Andrej Karpathy's autoresearch — an autonomous AI research loop where an AI agent iterates on a tiny language model's training code overnight, running ~100 experiments while you sleep.

Concept

Human writes program.md (the "research org" instructions)
    ↓
AI agent reads program.md
    ↓
Agent modifies train.py (model, optimizer, hyperparameters)
    ↓
Runs 5-minute training on GPU → measures val_bpb
    ↓
If improved → git commit (keep). If not → git revert (discard).
    ↓
Repeat (~12 experiments/hour, ~100 overnight)

Key insight: You don't touch any Python files. Instead, you program program.md — the Markdown instructions that guide the AI agent. You're programming the research organization, not running individual experiments.

Prerequisites

Hardware

Check your Mac chip: Apple menu → About This Mac → look for "Chip: M1/M2/M3/M4". Any Mac bought since late 2020 has Apple Silicon.

Software

Setup

Step 1: Install prerequisites

# Install uv (handles Python + all dependencies automatically)
curl -LsSf https://astral.sh/uv/install.sh | sh

# IMPORTANT: Close and reopen your terminal after installing uv

# Verify
uv --version
git --version

Step 2: Clone the repo

Mac (Apple Silicon):

cd ~/Desktop
git clone https://github.com/miolini/autoresearch-macos.git
cd autoresearch-macos

Linux/Windows (NVIDIA GPU):

cd ~/Desktop
git clone https://github.com/karpathy/autoresearch.git
cd autoresearch

About the Mac fork: Karpathy himself links to miolini/autoresearch-macos from his README. The developer (Artem Andreenko) has 167 public projects on GitHub and a years-long track record. The fork swaps FlashAttention-3 for PyTorch's built-in SDPA and adds Apple Metal/MPS optimizations. The entire codebase is ~630 lines — fully auditable in 20 minutes.

Step 3: Install dependencies and prepare data

# Install Python + all packages
uv sync

# Download training data + build tokenizer (one-time, ~2 min)
uv run prepare.py

# Run one test training to verify setup (~5 min)
uv run train.py

If the test training finishes and shows a val_bpb score — you're ready.

Step 4: Launch the autonomous loop

# Navigate to the project
cd ~/Desktop/autoresearch-macos  # or autoresearch on Linux

# Launch Claude Code
claude

Then type this prompt:

Hi have a look at program.md and let's kick off a new experiment! Let's do the setup first.

That's it. Minimize the window and go to sleep. The agent will:

1. Read program.md
2. Modify train.py with an experimental change
3. Run a 5-minute training
4. Check val_bpb — if improved, git commit; if not, git revert
5. Repeat all night

Pro tip: To make it fully autonomous, tell the agent upfront: "Run fully autonomously. Don't ask for confirmation between experiments. Keep going until I come back."

Project Structure

autoresearch/
├── prepare.py      ← Constants, data prep, runtime utilities (DO NOT modify)
├── train.py        ← Model + optimizer + training loop (agent modifies this)
├── program.md      ← Agent instructions (human modifies this)
├── pyproject.toml  ← Dependencies
├── results.tsv     ← Experiment log (score, memory, kept/discarded)
└── analysis.ipynb  ← Graphs showing progress over time

The three files that matter

Key Terminology

Design Choices

• Single file to modify. The agent only touches train.py. Keeps scope manageable and diffs reviewable.
• Fixed time budget. Training always runs for exactly 5 minutes, regardless of platform. This makes experiments directly comparable and means autoresearch finds the most optimal model for your specific hardware.
• Self-contained. No external dependencies beyond PyTorch. No distributed training, no complex configs. One GPU, one file, one metric.

Tips for Best Results

1. Start simple. Get one manual uv run train.py working first. If that doesn't work, the autonomous loop won't either.

2. Your one job is to improve program.md. Add instructions like:

   • "Try small improvements first"
   • "Focus on making val_bpb go down"
   • "Think step by step and explain every change before making it"
   • "If an experiment direction hasn't worked after 3 attempts, try something completely different"

3. Don't panic when experiments fail. Most will not improve the score. Out of 100 overnight experiments, maybe 10–20 are keepers. This is normal — the agent automatically keeps wins and discards losses.

4. Check in periodically at first. Watch the first 3–4 experiments to make sure the loop is working before going AFK.

5. More memory helps. 32 GB+ unified memory on Mac lets the agent explore larger models and more complex architectures.

Tuning for Smaller Hardware

If running on a Mac with limited memory, Karpathy recommends these adjustments (ask the agent to make them, or edit train.py/prepare.py yourself):

Checking Results

After a night of experiments:

# See the git history of successful experiments
git log --oneline

# Check the results log
cat results.tsv

# Open the analysis notebook (optional)
# Use Jupyter or Cursor to view analysis.ipynb

You'll find:

• Git history — each commit is a successful experiment that improved val_bpb
• Lower val_bpb — the model is genuinely smarter (baseline starts around 0.9979)
• Modified train.py — architecture tweaks, optimizer changes, hyperparameter adjustments
• results.tsv — every experiment with score, memory usage, and keep/discard status

Alternative Agent Options

Troubleshooting

References