Preprint · 2025 · Open access
ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search
arXiv (Cornell University) · Published 27 August 2025 · DOI 10.48550/arxiv.2508.19576
In brief
“ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search” is a 2025 preprint by Sining Zhoubian and 2 co-authors, published in arXiv (Cornell University). OpenAlex counts 11 citations as of 15 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).
- Citations
- 11
- Most cited in
- 2026 (11)
- References
- 37
- Authors
- 3
Abstract
With respect to improving the reasoning accuracy of LLMs, the representative reinforcement learning (RL) method - Group Relative Policy Optimization (GRPO) - has achieved critical success, yet it still suffers from the issue of insignificant reward signals. This paper introduces ReST-RL, a unified Reinforced Self-Training (ReST) policy-value framework that reconnects policy optimization and value-guided search to improve LLM reasoning ability. Firstly, ReST-GRPO adopts an optimized ReST-style algorithm to reshape the policy-induced trajectory distribution by increasing the reward variance of GRPO sampling and exposing the policy to more informative partial states, thereby improving training efficiency and effectiveness. Then, we further introduce a decoding optimization method, VM-MCTS, which trains a Value Model (VM) from self-collected Monte-Carlo Tree Search (MCTS) targets and deploys it through an adapted MCTS algorithm to provide precise process signals and verification scores, further enhancing reasoning accuracy. These two stages are internally dependent - ReST-GRPO yields higher-quality trajectories for value learning with VM-MCTS, which in turn enables more effective inference-time search. We validate our framework on multiple coding benchmarks (e.g., APPS, BigCodeBench, and HumanEval), where it significantly outperforms other reinforcement training baselines (naive GRPO, DAPO, and ReST-DPO), as well as decoding and verification baselines (e.g., PRM-BoN and ORM-MCTS), indicating its power to strengthen LLM reasoning capability. Moreover, we further evaluate ReST-RL on out-of-domain math and science reasoning tasks, where it achieves improved performance without target-domain tuning and favorable end-to-end efficiency trade-offs, providing preliminary transfer evidence beyond our primary coding domain.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Zhoubian, S., Zhang, D., & Tang, J. (2025). ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2508.19576
MLA 9
Zhoubian, Sining, et al. “ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search.” arXiv (Cornell University), 2025. https://doi.org/10.48550/arxiv.2508.19576.
Chicago
Zhoubian, Sining, Dan Zhang, and Jie Tang. 2025. “ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2508.19576.
Harvard
Zhoubian, S., Zhang, D. and Tang, J. (2025) ‘ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search’, arXiv (Cornell University). doi:10.48550/arxiv.2508.19576.
IEEE
S. Zhoubian, D. Zhang, and J. Tang, “ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search,” arXiv (Cornell University), 2025, doi: 10.48550/arxiv.2508.19576.
BibTeX
@misc{zhoubian2025rest,
title = {{ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search}},
author = {Zhoubian, Sining and Zhang, Dan and Tang, Jie},
howpublished = {arXiv (Cornell University)},
year = {2025},
doi = {10.48550/arxiv.2508.19576},
url = {https://doi.org/10.48550/arxiv.2508.19576}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- LLM reasoning
- reinforcement learning from human feedback
- Group Relative Policy Optimization
- self-training
- Monte Carlo tree search
- value model
- reward variance
- transfer learning
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Zhoubian, S., Zhang, D., & Tang, J. (2025). ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2508.19576
How many times has this paper been cited?
11 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at arXiv (Cornell University).
Who wrote this paper?
Sining Zhoubian, Dan Zhang, Jie Tang.
What is the DOI of this paper?
10.48550/arxiv.2508.19576. The permanent link is https://doi.org/10.48550/arxiv.2508.19576.
Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.