Preprint · 2025 · Open access

Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning

Liu, Qihao, Ye, Luoxin, Ma, Wufei, Chou, Yu-Cheng, Alan Yuille

arXiv (Cornell University) · Published 18 December 2025 · DOI 10.48550/arxiv.2512.16917

Free PDFView at the publisherCite this paper

In brief

“Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning” is a 2025 preprint by Liu, Qihao and 4 co-authors, published in arXiv (Cornell University). OpenAlex counts 6 citations as of 15 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).

Citations
6
Most cited in
2026 (6)
References
31
Authors
5

Abstract

Large language models (LLMs) with explicit reasoning capabilities excel at mathematical reasoning yet still commit process errors, such as incorrect calculations, brittle logic, and superficially plausible but invalid steps. In this paper, we introduce Generative Adversarial Reasoner, an on-policy joint training framework designed to enhance reasoning by co-evolving an LLM reasoner and an LLM-based discriminator through adversarial reinforcement learning. A compute-efficient review schedule partitions each reasoning chain into logically complete slices of comparable length, and the discriminator evaluates each slice's soundness with concise, structured justifications. Learning couples complementary signals: the LLM reasoner is rewarded for logically consistent steps that yield correct answers, while the discriminator earns rewards for correctly detecting errors or distinguishing traces in the reasoning process. This produces dense, well-calibrated, on-policy step-level rewards that supplement sparse exact-match signals, improving credit assignment, increasing sample efficiency, and enhancing overall reasoning quality of LLMs. Across various mathematical benchmarks, the method delivers consistent gains over strong baselines with standard RL post-training. Specifically, on AIME24, we improve DeepSeek-R1-Distill-Qwen-7B from 54.0 to 61.3 (+7.3) and DeepSeek-R1-Distill-Llama-8B from 43.7 to 53.7 (+10.0). The modular discriminator also enables flexible reward shaping for objectives such as teacher distillation, preference alignment, and mathematical proof-based reasoning.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Qihao, L., Luoxin, Y., Wufei, M., Yu-Cheng, C., & Yuille, A. (2025). Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2512.16917

MLA 9

Qihao, Liu,, et al. “Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning.” arXiv (Cornell University), 2025. https://doi.org/10.48550/arxiv.2512.16917.

Chicago

Qihao, Liu,, Ye, Luoxin, Ma, Wufei, Chou, Yu-Cheng, and Alan Yuille. 2025. “Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2512.16917.

Harvard

Qihao, L. et al. (2025) ‘Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning’, arXiv (Cornell University). doi:10.48550/arxiv.2512.16917.

IEEE

L. Qihao, Y. Luoxin, M. Wufei, C. Yu-Cheng, and A. Yuille, “Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning,” arXiv (Cornell University), 2025, doi: 10.48550/arxiv.2512.16917.

BibTeX

@misc{qihao2025generative,
  title     = {{Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning}},
  author    = {Qihao, Liu, and Luoxin, Ye, and Wufei, Ma, and Yu-Cheng, Chou, and Yuille, Alan},
  howpublished = {arXiv (Cornell University)},
  year      = {2025},
  doi       = {10.48550/arxiv.2512.16917},
  url       = {https://doi.org/10.48550/arxiv.2512.16917}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling

  • large language model reasoning
  • adversarial reinforcement learning
  • mathematical reasoning
  • credit assignment
  • reward shaping
  • preference alignment

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Qihao, L., Luoxin, Y., Wufei, M., Yu-Cheng, C., & Yuille, A. (2025). Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2512.16917

How many times has this paper been cited?

6 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at arXiv (Cornell University).

Who wrote this paper?

Liu, Qihao, Ye, Luoxin, Ma, Wufei, Chou, Yu-Cheng, Alan Yuille.

What is the DOI of this paper?

10.48550/arxiv.2512.16917. The permanent link is https://doi.org/10.48550/arxiv.2512.16917.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.