Label-free RL for reasoning

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

Training solely on self-generated feedback can reinforce existing biases and lead to training collapse. Can unsupervised reasoning emerge through cooperative multi-agent training?

Yunhao Yang^1,* Yuexin Bian^2,* Yunjie Tian⁴ Di Fu⁴ Tianjin Huang³ Yuanyuan Shi² Ziang Xiao¹ Nuno Vasconcelos² Yijiang Li^2,†

¹ Johns Hopkins University · ² UC San Diego · ³ University of Exeter · ⁴ Independent Researcher

* equal contribution · † project lead

Paper Code BibTeX

01 / The problem

RLVR's strongest successes still depend heavily on ground-truth supervision.

Reinforcement learning with verifiable rewards has emerged as a powerful approach for improving reasoning in language and vision-language models. Such supervision is costly to obtain and becomes increasingly scarce as target reasoning capabilities approach or surpass what humans can reliably evaluate.

02 / The idea

Co-RL takes the reward from a peer, never from the model itself.

Independently trained models make different errors. Each one can therefore correct mistakes the other cannot see in its own generations. Co-RL turns this into supervision, and each agent is rewarded when its answer matches a peer's majority vote.

Comparison of four label-free reward designs: TTRL and Co-rewarding derive rewards from self-generated agreement, CoMAS scores multi-turn interactions with one of its own agents as judge, and Co-RL derives rewards directly from peer votes, passed along a directed ring beyond two agents. — Where the reward comes from in four label-free designs. Beyond two agents, the votes pass along a directed ring. Click to enlarge.

STEP 01 Sample and vote

Each agent samples several completions for the same unlabeled prompt and reduces them to a majority vote.

STEP 02 Reward against the peer

A completion earns reward 1 when its answer matches the peer's vote. An agent never contributes to its own supervision target.

STEP 03 Update independently

GRPO updates each policy separately. The agents share no parameters and exchange no gradients.

Overview of Co-RL with two agents: each agent samples K responses to the same unlabeled question, forms a majority-vote pseudo-label, and each rollout is rewarded by agreement with the peer's pseudo-label, followed by independent GRPO updates. — The full mechanism with two agents. Each agent samples K responses, votes, and is rewarded by agreement with its peer's vote. Click to enlarge.

03 / Results

Co-RL trains without labels and matches GRPO trained with them.

GT-Reward is the same recipe with the same models, data, and budget, plus the ground truth. Co-RL matches or surpasses it in 11 of 14 settings, and matches or beats TTRL in 13 of 14. Same family trains two agents initialized from the same base model, Different family pairs one agent from each of two families, and Different family+ further decouples the training data.

Average gain over the Base model for every setting: Co-RL and TTRL as paired bars, GT-Reward as a dashed reference tick. Panels: text two-agent pairs, text three-agent ring, vision-language. — Average gain over the Base model on each setting's benchmark suite (%). Dashed ticks are GT-Reward, the labeled reference. Click to enlarge.

Apart from AMC, every number is a single run evaluated single-sample. Settings where Co-RL falls below the reference are shown as they are. MMR1 rows use the corrected multiple-choice grader and open-r1 rows the legacy grader, so the two are never compared with each other.

Full per-benchmark results

Pick a domain and a backbone to see the full table. Violet cells sit above the Base model and red cells below. Bold is the best label-free result and underline the second best, as in the paper.

Domain Backbone

Co-RL also trains stably where self-rewarding degenerates

The tables above show endpoints. The curves show the whole run at four scales.

(a)Several self-rewarding runs collapse or diverge midway. Co-RL and the labeled reference hold their accuracy.
(b)RENT drives the reward variation toward zero. Co-RL keeps it steady for the whole run.
(c)RENT shows sharp increases in completion length and Intuitor degenerates on some models. Co-RL stays flat.

Training dynamics at four model scales: validation accuracy, within-group reward standard deviation, and mean completion length for Co-RL, GT-Reward, TTRL, RENT, and Intuitor. Self-rewarding baselines show reward collapse, length degeneration, or divergence. Co-RL stays stable. — Validation accuracy, reward standard deviation, and mean completion length at four scales. Runs marked diverged leave the plotted range. Click to enlarge.

04 / Why it works

What a peer teaches depends on how different its mistakes are.

Two similar models make the same mistakes and can reinforce the same wrong answers. The paper pushes diversity through three channels, and each one is measurable.

Decoupled policy optimization

Each policy keeps its own parameters and optimizer state. The only interaction is the exchanged vote.

(a)Different families keep lower agreement during pseudo-labeling, and the two agents never collapse into one voice.
(b)Crossing families yields more accurate pseudo-labels, so the extra disagreement is not extra noise.
(c)Downstream accuracy is higher, and training stays stable relative to self-rewarding.

Three panels for the 7B pair during training: inter-model agreement, pseudo-label accuracy, and MATH-500 evaluation accuracy for Co-RL same family, Co-RL different family, and TTRL. — (a) Agreement, (b) pseudo-label accuracy, and (c) MATH-500 accuracy during training, for the 7B pair. Click to enlarge.

Different model families and sizes

Families differ in architecture, tokenization, and pretraining data, and vision-language families also differ in their vision encoders. How much two models' errors overlap can be measured before any training, on 500 MATH problems per pair.

Vertical bar chart of error overlap for twelve base-checkpoint pairs: every different-family pair ends at or below 0.42, every same-family and seed-only pair at or above 0.51, and the shaded strip between is empty. — Error overlap κ (Cohen's kappa) for twelve base-checkpoint pairs, before any RL. Click to enlarge.

The finding Every different-family pair ends at or below κ 0.42 and every same-family or seed-only pair at or above κ 0.51. No pair lands between the two groups. Crossing families lowers the average overlap from 0.53 to 0.38, and the downstream gains in section 03 follow the same ordering.

Decoupled input formation

Different family+ adds the third channel. One agent trains on the original MATH prompt and the other on a DeepSeek-V3 rewrite, so the pair no longer shares its training data. The rewrite recasts the problem in a new scenario and never changes the answer.

Original MATH prompt

How many vertical asymptotes does the graph of y = 2/(x²+x−6) have?

Rephrased

The function f(t) = 2/(t²+t−6) describes the temperature of a chemical reaction over time t. How many vertical asymptotes appear on the graph of this function?

Answer unchanged · 2

05 / Ablations

Three deeper ablations

Matched budgets

Ensembling two TTRL models does not close the gap

The same two base models are trained with TTRL and ensembled the same way, maj@8 on three math benchmarks. Co-RL's ensemble reaches 66.9% versus 64.9%, and the TTRL ensemble stays below its own best single model.

Vertical grouped bars of macro-average maj@8 on a truncated axis: Co-RL beats TTRL for both single models and for the ensemble, and the TTRL ensemble scores below its best single model.

Open the paper

Versus multi-agent RL

Co-RL beats CoMAS under CoMAS's own protocol

Same setup, official implementation, and graders, with prior rows quoted from their paper. Co-RL reaches 62.97% versus 58.94% for CoMAS, with half the agents and no judge.

Vertical bars of seven-benchmark averages under the CoMAS protocol: Base 56.92, MAPoRL 58.22, TTRL 58.18, CoMAS 58.94, Co-RL 62.97.

Open the paper

Three agents

One ring trains three models, and all three improve

Qwen2.5-3B, Llama-3.2-3B-Instruct, and Qwen3-1.7B train together in one ring. They gain 7.8%, 6.0%, and 8.2% and match or beat their labeled references.

Vertical grouped bars from Base to Co-RL for the three ring-trained models on a truncated axis, with dashed GT-Reward marks: gains of 7.8, 6.0, and 8.2 points.

Full rows in the explorer

06 / Code

The code is public.

The release is Apache-2.0. Every setting on this page has a launcher in the repository, with the baselines beside the method.

BibTeX

@article{yang2026corl,
  title   = {Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL},
  author  = {Yang, Yunhao and Bian, Yuexin and Tian, Yunjie and Fu, Di and Huang, Tianjin and Shi, Yuanyuan and Xiao, Ziang and Vasconcelos, Nuno and Li, Yijiang},
  journal = {arXiv preprint arXiv:2608.17253},
  year    = {2026},
  eprint  = {2608.17253},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG}
}