Preprint · 2025 · Open access

ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search

Sining Zhoubian, Dan Zhang, Jie Tang

arXiv (Cornell University) · Published 27 August 2025 · DOI 10.48550/arxiv.2508.19576

Free PDFView at the publisherCite this paper

In brief

“ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search” is a 2025 preprint by Sining Zhoubian and 2 co-authors, published in arXiv (Cornell University). OpenAlex counts 11 citations as of 15 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).

Citations
11
Most cited in
2026 (11)
References
37
Authors
3

Abstract

With respect to improving the reasoning accuracy of LLMs, the representative reinforcement learning (RL) method - Group Relative Policy Optimization (GRPO) - has achieved critical success, yet it still suffers from the issue of insignificant reward signals. This paper introduces ReST-RL, a unified Reinforced Self-Training (ReST) policy-value framework that reconnects policy optimization and value-guided search to improve LLM reasoning ability. Firstly, ReST-GRPO adopts an optimized ReST-style algorithm to reshape the policy-induced trajectory distribution by increasing the reward variance of GRPO sampling and exposing the policy to more informative partial states, thereby improving training efficiency and effectiveness. Then, we further introduce a decoding optimization method, VM-MCTS, which trains a Value Model (VM) from self-collected Monte-Carlo Tree Search (MCTS) targets and deploys it through an adapted MCTS algorithm to provide precise process signals and verification scores, further enhancing reasoning accuracy. These two stages are internally dependent - ReST-GRPO yields higher-quality trajectories for value learning with VM-MCTS, which in turn enables more effective inference-time search. We validate our framework on multiple coding benchmarks (e.g., APPS, BigCodeBench, and HumanEval), where it significantly outperforms other reinforcement training baselines (naive GRPO, DAPO, and ReST-DPO), as well as decoding and verification baselines (e.g., PRM-BoN and ORM-MCTS), indicating its power to strengthen LLM reasoning capability. Moreover, we further evaluate ReST-RL on out-of-domain math and science reasoning tasks, where it achieves improved performance without target-domain tuning and favorable end-to-end efficiency trade-offs, providing preliminary transfer evidence beyond our primary coding domain.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Zhoubian, S., Zhang, D., & Tang, J. (2025). ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2508.19576

MLA 9

Zhoubian, Sining, et al. “ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search.” arXiv (Cornell University), 2025. https://doi.org/10.48550/arxiv.2508.19576.

Chicago

Zhoubian, Sining, Dan Zhang, and Jie Tang. 2025. “ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2508.19576.

Harvard

Zhoubian, S., Zhang, D. and Tang, J. (2025) ‘ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search’, arXiv (Cornell University). doi:10.48550/arxiv.2508.19576.

IEEE

S. Zhoubian, D. Zhang, and J. Tang, “ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search,” arXiv (Cornell University), 2025, doi: 10.48550/arxiv.2508.19576.

BibTeX

@misc{zhoubian2025rest,
  title     = {{ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search}},
  author    = {Zhoubian, Sining and Zhang, Dan and Tang, Jie},
  howpublished = {arXiv (Cornell University)},
  year      = {2025},
  doi       = {10.48550/arxiv.2508.19576},
  url       = {https://doi.org/10.48550/arxiv.2508.19576}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Computer Science › Information Systems › Software Engineering Research

  • LLM reasoning
  • reinforcement learning from human feedback
  • Group Relative Policy Optimization
  • self-training
  • Monte Carlo tree search
  • value model
  • reward variance
  • transfer learning

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Zhoubian, S., Zhang, D., & Tang, J. (2025). ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2508.19576

How many times has this paper been cited?

11 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at arXiv (Cornell University).

Who wrote this paper?

Sining Zhoubian, Dan Zhang, Jie Tang.

What is the DOI of this paper?

10.48550/arxiv.2508.19576. The permanent link is https://doi.org/10.48550/arxiv.2508.19576.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.