AMD/ROCm fork of karpathy/autoresearch — autonomous pretraining research on consumer RDNA4 GPUs.
This fork ports autoresearch to run on AMD GPUs with ROCm. Key changes:
- FlashAttention-3 replaced with PyTorch native SDPA (
F.scaled_dot_product_attention) - PyTorch nightly ROCm 7.2 wheels (stable wheels lack gfx1201 support)
- HIP caching allocator disabled — workaround for memory access faults on RDNA4
torch.compiledefault off — Triton compilation overhead eats into the 5-min budget- Model scaled for 16GB VRAM — DEPTH=6, BATCH_SIZE=16 (~26M params)
| Component | Version |
|---|---|
| GPU | AMD Radeon RX 9070 XT (gfx1201, RDNA4, 16GB) |
| OS | CachyOS (Arch-based), kernel 6.19+ |
| ROCm | 7.2.0 (system packages) |
| PyTorch | 2.12.0.dev nightly (rocm7.2 wheels) |
val_bpb: 1.758637
training_seconds: 306.0
num_steps: 43
num_params_M: 26.3
mfu_percent: 11.08
Requirements: AMD GPU with ROCm 7.2+ support, Python 3.10, uv.
# CachyOS / Arch
sudo pacman -S --needed rocm-hip-sdk rocminfo rocm-smi-lib
# Verify
/opt/rocm/bin/rocminfo | grep gfx
/opt/rocm/bin/rocm-smigit clone https://github.com/ryancatlin/autoresearch-amd.git
cd autoresearch-amd
uv syncuv run prepare.pyuv run train.pyPoint Claude Code (or similar) at this repo:
Hi have a look at program.md and let's kick off a new experiment! let's do the setup first.
| Issue | Workaround |
|---|---|
| HIP caching allocator causes GPU memory faults on gfx1201 | PYTORCH_NO_HIP_MEMORY_CACHING=1 (set in train.py) |
torch.compile Triton compilation is slow |
Default off; set AUTORESEARCH_COMPILE=1 to enable |
| Stable PyTorch ROCm wheels lack gfx1201 | Must use nightly rocm7.2 wheels |
peak_vram_mb reports 0.0 |
Side effect of disabling HIP caching allocator |
(null): No such file or directory warning |
Cosmetic; missing amdgpu.ids file, harmless |
prepare.py — constants, data prep + runtime utilities (do not modify)
train.py — model, optimizer, training loop (agent modifies this)
program.md — agent instructions (includes AMD hardware constraints)
pyproject.toml — dependencies (ROCm PyTorch nightly)
See upstream README for full project context and design choices.
MIT