Reward-Adaptive Co-Evolution of Policy and Rubric

RACE (Reward-Adaptive Co-Evolution) is a framework for rubric-based reinforcement learning in which the policy and the rubric co-evolve. An operator should change two things: (1) which rollout groups feed policy updates, and (2) when to switch into and out of rubric optimization. Both are governed by thresholds that adapt online rather than by a globally prescribed schedule.

Implementation details

• At the group level, per-rollout-group training-signal routing classifies each group as informative, diversity-collapsed, or rubric-failed; the policy is updated only with informative groups, and rubric failures are cached.
• At the phase level, event-triggered phase switching enters rubric optimization upon persistent reward saturation and returns to policy training when rubric-reward improvement plateaus.
• Rubric phases train the generator on cached pairs using a frozen judge and a scale-normalized pairwise-margin reward.

Known limitations

• Static rubrics can lose discriminative power as the policy distribution evolves.

Research · arXiv cs.AI · Publisher website: arxiv.org · Last verified 3 November 2026 · Catalog ID 128456