Preprint · 2025 · Open access
On the Emergence of Thinking in LLMs I: Searching for the Right Intuition
arXiv (Cornell University) · Published 10 February 2025 · DOI 10.48550/arxiv.2502.06773
In brief
“On the Emergence of Thinking in LLMs I: Searching for the Right Intuition” is a 2025 preprint by Guanghao Ye and 7 co-authors, published in arXiv (Cornell University). OpenAlex counts 21 citations as of 15 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).
- Citations
- 21
- Most cited in
- 2025 (15)
- References
- 55
- Authors
- 8
Abstract
Recent AI advancements, such as OpenAI's new models, are transforming LLMs into LRMs (Large Reasoning Models) that perform reasoning during inference, taking extra time and compute for higher-quality outputs. We aim to uncover the algorithmic framework for training LRMs. Methods like self-consistency, PRM, and AlphaZero suggest reasoning as guided search. We ask: what is the simplest, most scalable way to enable search in LLMs? We propose a post-training framework called Reinforcement Learning via Self-Play (RLSP). RLSP involves three steps: (1) supervised fine-tuning with human or synthetic demonstrations of the reasoning process, (2) using an exploration reward signal to encourage diverse and efficient reasoning behaviors, and (3) RL training with an outcome verifier to ensure correctness while preventing reward hacking. Our key innovation is to decouple exploration and correctness signals during PPO training, carefully balancing them to improve performance and efficiency. Empirical studies in the math domain show that RLSP improves reasoning. On the Llama-3.1-8B-Instruct model, RLSP can boost performance by 23% in MATH-500 test set; On AIME 2024 math problems, Qwen2.5-32B-Instruct improved by 10% due to RLSP. However, a more important finding of this work is that the models trained using RLSP, even with the simplest exploration reward that encourages the model to take more intermediate steps, showed several emergent behaviors such as backtracking, exploration of ideas, and verification. These findings demonstrate that RLSP framework might be enough to enable emergence of complex reasoning abilities in LLMs when scaled. Lastly, we propose a theory as to why RLSP search strategy is more suitable for LLMs inspired by a remarkable result that says CoT provably increases computational power of LLMs, which grows as the number of steps in CoT \cite{li2024chain,merrill2023expresssive}.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Ye, G., Pham, K. D., Zhang, X., Gopi, S., Peng, B., Li, B., Kulkarni, J., & Inan, H. A. (2025). On the Emergence of Thinking in LLMs I: Searching for the Right Intuition. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2502.06773
MLA 9
Ye, Guanghao, et al. “On the Emergence of Thinking in LLMs I: Searching for the Right Intuition.” arXiv (Cornell University), 2025. https://doi.org/10.48550/arxiv.2502.06773.
Chicago
Ye, Guanghao, Khiem Duc Pham, Xinzhi Zhang, Sivakanth Gopi, Baolin Peng, Beibin Li, Janardhan Kulkarni, and Huseyin A. Inan. 2025. “On the Emergence of Thinking in LLMs I: Searching for the Right Intuition.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2502.06773.
Harvard
Ye, G. et al. (2025) ‘On the Emergence of Thinking in LLMs I: Searching for the Right Intuition’, arXiv (Cornell University). doi:10.48550/arxiv.2502.06773.
IEEE
G. Ye et al., “On the Emergence of Thinking in LLMs I: Searching for the Right Intuition,” arXiv (Cornell University), 2025, doi: 10.48550/arxiv.2502.06773.
BibTeX
@misc{ye2025emergence,
title = {{On the Emergence of Thinking in LLMs I: Searching for the Right Intuition}},
author = {Ye, Guanghao and Pham, Khiem Duc and Zhang, Xinzhi and Gopi, Sivakanth and Peng, Baolin and Li, Beibin and Kulkarni, Janardhan and Inan, Huseyin A.},
howpublished = {arXiv (Cornell University)},
year = {2025},
doi = {10.48550/arxiv.2502.06773},
url = {https://doi.org/10.48550/arxiv.2502.06773}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- large reasoning models
- LLM reasoning
- guided search
- reward hacking
- PPO
- chain-of-thought prompting
- emergent behavior
- backtracking
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Ye, G., Pham, K. D., Zhang, X., Gopi, S., Peng, B., Li, B., Kulkarni, J., & Inan, H. A. (2025). On the Emergence of Thinking in LLMs I: Searching for the Right Intuition. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2502.06773
How many times has this paper been cited?
21 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at arXiv (Cornell University).
Who wrote this paper?
Guanghao Ye, Khiem Duc Pham, Xinzhi Zhang, Sivakanth Gopi, Baolin Peng, Beibin Li, Janardhan Kulkarni, Huseyin A. Inan.
What is the DOI of this paper?
10.48550/arxiv.2502.06773. The permanent link is https://doi.org/10.48550/arxiv.2502.06773.
Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.