Conference paper · 2026 · Open access
FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models
Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP), 12204–12241 · Published 1 January 2026 · DOI 10.18653/2026.findings-emnlp.579
In brief
“FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models” is a 2026 conference paper by Zhouliang Yu and 12 co-authors, published in Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP). OpenAlex counts 85 citations as of 21 November 2026, which puts it in the top 1% of works published in 2026 by citations. It is open access under a CC BY-NC-SA licence: the full text is free to read at Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP).
- Citations
- 85
- Most cited in
- 2026 (65)
- Field-weighted impact
- 87.09
- References
- 41
- Authors
- 13
Abstract
We introduce FormalMATH, a large-scale Lean4 benchmark for formal mathematical reasoning, comprising 5,560 semantically validated Lean4 statements spanning competitionlevel and undergraduate mathematics.To improve reliability, FormalMATH is constructed through a human-in-the-loop pipeline that combines type checking, LLM-based semantic verification against the informal-formal statements alignment, negation-based disproving, and expert review.We conduct a systematic evaluation of frontier theorem provers under test-time scaling on FormalMATH.Results show that current provers remain far from solving the benchmark: the strongest model, DeepSeek-Prover-V2 (671B), achieves only 28.31% Pass@32.We further observe substantial domain imbalance across provers, with stronger performance in applied mathematics and algebra but weaker results in discrete mathematics.We identify common failure modes in current provers, including incomplete proofs, weak handling of inequalities, misuse of automation tactics, and redundant hypothesis introduction.Extreme test-time scaling (e.g., Pass@3200) yields only modest gains on a subset of FormalMATH, suggesting that current training paradigms do not equip provers to effectively exploit additional sampling budget.We also observe a counterintuitive negative effect when provers are given humanwritten natural-language solution references: such references can measurably degrade formal proof generation rather than aid it.Overall, FormalMATH provides a challenging and reliable testbed for measuring progress in formal mathematical reasoning.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Yu, Z., Peng, R., Ding, K., Li, Y., Peng, Z., Liu, M., Zhang, Y. M., Yuan, Z., Xin, H., Huang, W., Wen, Y., Zhang, G., & Liu, W. (2026). FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models. Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP), 12204–12241. https://doi.org/10.18653/2026.findings-emnlp.579
MLA 9
Yu, Zhouliang, et al. “FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models.” Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP), 2026, pp. 12204–12241. https://doi.org/10.18653/2026.findings-emnlp.579.
Chicago
Yu, Zhouliang, Ruotian Peng, Keyi Ding, Yizhe Li, Zhongyuan Peng, Minghao Liu, Y. M. Zhang, et al. 2026. “FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models.” Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP): 12204–12241. https://doi.org/10.18653/2026.findings-emnlp.579.
Harvard
Yu, Z. et al. (2026) ‘FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models’, Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP), pp. 12204–12241. doi:10.18653/2026.findings-emnlp.579.
IEEE
Z. Yu et al., “FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models,” Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP), pp. 12204–12241, 2026, doi: 10.18653/2026.findings-emnlp.579.
BibTeX
@inproceedings{yu2026formalmath,
title = {{FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models}},
author = {Yu, Zhouliang and Peng, Ruotian and Ding, Keyi and Li, Yizhe and Peng, Zhongyuan and Liu, Minghao and Zhang, Y. M. and Yuan, Zheng and Xin, Huajian and Huang, Wenhao and Wen, Yandong and Zhang, Ge and Liu, Weiyang},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)},
year = {2026},
pages = {12204--12241},
doi = {10.18653/2026.findings-emnlp.579},
url = {https://doi.org/10.18653/2026.findings-emnlp.579}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Lean 4
- theorem provers
- benchmarking
- large language models
- formal verification
- human-in-the-loop
- test-time scaling
- proof generation
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Yu, Z., Peng, R., Ding, K., Li, Y., Peng, Z., Liu, M., Zhang, Y. M., Yuan, Z., Xin, H., Huang, W., Wen, Y., Zhang, G., & Liu, W. (2026). FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models. Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP), 12204–12241. https://doi.org/10.18653/2026.findings-emnlp.579
How many times has this paper been cited?
85 times, according to OpenAlex on 21 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP): sign in and open the PDF from this page.
Who wrote this paper?
Zhouliang Yu, Ruotian Peng, Keyi Ding, Yizhe Li, Zhongyuan Peng, Minghao Liu, Y. M. Zhang, Zheng Yuan, Huajian Xin, Wenhao Huang, Yandong Wen, Ge Zhang and 1 more.
What is the DOI of this paper?
10.18653/2026.findings-emnlp.579, permanently at https://doi.org/10.18653/2026.findings-emnlp.579. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 21 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.