Preprint · 2025 · Open access

Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search

Maohao Shen, Guangtao Zeng, Zhenting Qi, Zhang-Wei Hong, Zhenfang Chen, Wei Lu, Gregory W. Wornell, Subhro Das, David Cox, Chuang Gan

arXiv (Cornell University) · Published 4 February 2025 · DOI 10.48550/arxiv.2502.02508

Free PDFView at the publisherCite this paper

In brief

“Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search” is a 2025 preprint by Maohao Shen and 9 co-authors, published in arXiv (Cornell University). OpenAlex counts 53 citations as of 15 November 2026. It is open access under a CC BY-NC-ND licence: the full text is free to read at arXiv (Cornell University).

Citations
53
Most cited in
2025 (29)
References
41
Authors
10

Abstract

Large language models (LLMs) have demonstrated remarkable reasoning capabilities across diverse domains. Recent studies have shown that increasing test-time computation enhances LLMs' reasoning capabilities. This typically involves extensive sampling at inference time guided by an external LLM verifier, resulting in a two-player system. Despite external guidance, the effectiveness of this system demonstrates the potential of a single LLM to tackle complex tasks. Thus, we pose a new research problem: Can we internalize the searching capabilities to fundamentally enhance the reasoning abilities of a single LLM? This work explores an orthogonal direction focusing on post-training LLMs for autoregressive searching (i.e., an extended reasoning process with self-reflection and self-exploration of new strategies). To achieve this, we propose the Chain-of-Action-Thought (COAT) reasoning and a two-stage training paradigm: 1) a small-scale format tuning stage to internalize the COAT reasoning format and 2) a large-scale self-improvement stage leveraging reinforcement learning. Our approach results in Satori, a 7B LLM trained on open-source models and data. Extensive empirical evaluations demonstrate that Satori achieves state-of-the-art performance on mathematical reasoning benchmarks while exhibits strong generalization to out-of-domain tasks. Code, data, and models are fully open-sourced.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Shen, M., Zeng, G., Qi, Z., Hong, Z.-W., Chen, Z., Lu, W., Wornell, G. W., Das, S., Cox, D., & Gan, C. (2025). Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2502.02508

MLA 9

Shen, Maohao, et al. “Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search.” arXiv (Cornell University), 2025. https://doi.org/10.48550/arxiv.2502.02508.

Chicago

Shen, Maohao, Guangtao Zeng, Zhenting Qi, Zhang-Wei Hong, Zhenfang Chen, Wei Lu, Gregory W. Wornell, Subhro Das, David Cox, and Chuang Gan. 2025. “Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2502.02508.

Harvard

Shen, M. et al. (2025) ‘Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search’, arXiv (Cornell University). doi:10.48550/arxiv.2502.02508.

IEEE

M. Shen et al., “Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search,” arXiv (Cornell University), 2025, doi: 10.48550/arxiv.2502.02508.

BibTeX

@misc{shen2025satori,
  title     = {{Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search}},
  author    = {Shen, Maohao and Zeng, Guangtao and Qi, Zhenting and Hong, Zhang-Wei and Chen, Zhenfang and Lu, Wei and Wornell, Gregory W. and Das, Subhro and Cox, David and Gan, Chuang},
  howpublished = {arXiv (Cornell University)},
  year      = {2025},
  doi       = {10.48550/arxiv.2502.02508},
  url       = {https://doi.org/10.48550/arxiv.2502.02508}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling

  • large language models
  • LLM reasoning
  • reinforcement learning
  • human development
  • mathematical reasoning
  • self-reflection
  • out-of-domain generalization

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Shen, M., Zeng, G., Qi, Z., Hong, Z.-W., Chen, Z., Lu, W., Wornell, G. W., Das, S., Cox, D., & Gan, C. (2025). Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2502.02508

How many times has this paper been cited?

53 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at arXiv (Cornell University).

Who wrote this paper?

Maohao Shen, Guangtao Zeng, Zhenting Qi, Zhang-Wei Hong, Zhenfang Chen, Wei Lu, Gregory W. Wornell, Subhro Das, David Cox, Chuang Gan.

What is the DOI of this paper?

10.48550/arxiv.2502.02508. The permanent link is https://doi.org/10.48550/arxiv.2502.02508.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.