Preprint · 2025 · Open access

RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning

Qianyue Hao, Sibo Li, Jian Yuan, Yong Li

arXiv (Cornell University) · Published 20 May 2025 · DOI 10.48550/arxiv.2505.14140

Free PDFView at the publisherCite this paper

In brief

“RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning” is a 2025 preprint by Qianyue Hao and 3 co-authors, published in arXiv (Cornell University). OpenAlex counts 22 citations as of 15 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).

Citations
22
Most cited in
2026 (12)
References
36
Authors
4

Abstract

Despite rapid advancements in large language models (LLMs), the token-level autoregressive nature constrains their complex reasoning capabilities. To enhance LLM reasoning, inference-time techniques, including Chain/Tree/Graph-of-Thought(s), successfully improve the performance, as they are fairly cost-effective by guiding reasoning through sophisticated logical structures without modifying LLMs' parameters. However, these manually predefined, task-agnostic frameworks are applied uniformly across diverse tasks, lacking adaptability. To improve this, we propose RL-of-Thoughts (RLoT), where we train a lightweight navigator model with reinforcement learning (RL) to adaptively enhance LLM reasoning at inference time. Specifically, we design five basic logic blocks from the perspective of human cognition. During the reasoning process, the trained RL navigator dynamically selects the suitable logic blocks and combines them into task-specific logical structures according to problem characteristics. Experiments across multiple reasoning benchmarks (AIME, MATH, GPQA, etc.) with multiple LLMs (GPT, Llama, Qwen, and DeepSeek) illustrate that RLoT outperforms established inference-time techniques by up to 13.4%. Remarkably, with less than 3K parameters, our RL navigator is able to make sub-10B LLMs comparable to 100B-scale counterparts. Moreover, the RL navigator demonstrates strong transferability: a model trained on one specific LLM-task pair can effectively generalize to unseen LLMs and tasks. Our code is open-source at https://github.com/tsinghua-fib-lab/RL-LLM-Reasoning for reproducibility.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Hao, Q., Li, S., Yuan, J., & Li, Y. (2025). RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2505.14140

MLA 9

Hao, Qianyue, et al. “RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning.” arXiv (Cornell University), 2025. https://doi.org/10.48550/arxiv.2505.14140.

Chicago

Hao, Qianyue, Sibo Li, Jian Yuan, and Yong Li. 2025. “RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2505.14140.

Harvard

Hao, Q. et al. (2025) ‘RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning’, arXiv (Cornell University). doi:10.48550/arxiv.2505.14140.

IEEE

Q. Hao, S. Li, J. Yuan, and Y. Li, “RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning,” arXiv (Cornell University), 2025, doi: 10.48550/arxiv.2505.14140.

BibTeX

@misc{hao2025thoughts,
  title     = {{RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning}},
  author    = {Hao, Qianyue and Li, Sibo and Yuan, Jian and Li, Yong},
  howpublished = {arXiv (Cornell University)},
  year      = {2025},
  doi       = {10.48550/arxiv.2505.14140},
  url       = {https://doi.org/10.48550/arxiv.2505.14140}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202411
202599
20261212

Topic and keywords

Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling

  • LLM reasoning
  • chain-of-thought
  • tree-of-thought
  • graph-of-thought
  • logic blocks
  • mathematical reasoning
  • question answering

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Hao, Q., Li, S., Yuan, J., & Li, Y. (2025). RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2505.14140

How many times has this paper been cited?

22 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at arXiv (Cornell University).

Who wrote this paper?

Qianyue Hao, Sibo Li, Jian Yuan, Yong Li.

What is the DOI of this paper?

10.48550/arxiv.2505.14140. The permanent link is https://doi.org/10.48550/arxiv.2505.14140.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.