Preprint · 2025 · Open access

Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark

Ziming Cheng, Bin Xu, Gong, Lisheng, Song, Zuhe, Tianshuo Zhou, Zhong, Shiqi, Siyu Ren, Mingxiang Chen, Xiangchao Meng, Yuxin Zhang, Yanlin Li, Lei Ren and 5 more

arXiv (Cornell University) · Published 4 June 2025 · DOI 10.48550/arxiv.2506.04280

Open PDFView at the publisherCite this paper

In brief

“Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark” is a 2025 preprint by Ziming Cheng and 16 co-authors, published in arXiv (Cornell University). OpenAlex counts 23 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).

Citations
23
Most cited in
2026 (22)
References
48
Authors
17

Abstract

With enhanced capabilities and widespread applications, Multimodal Large Language Models (MLLMs) are increasingly required to process and reason over multiple images simultaneously. However, existing MLLM benchmarks focus either on single-image visual reasoning or on multi-image understanding tasks with only final-answer evaluation, leaving the reasoning capabilities of MLLMs over multi-image inputs largely underexplored. To address this gap, we introduce the $\textbf{Multimodal Multi-image Reasoning Benchmark (MMRB)}$, the first benchmark designed to evaluate structured visual reasoning across multiple images. MMRB comprises $\textbf{92 sub-tasks}$ covering spatial, temporal, and semantic reasoning, with multi-solution, CoT-style annotations generated by GPT-4o and refined by human experts. A derivative subset is designed to evaluate multimodal reward models in multi-image scenarios. To support fast and scalable evaluation, we propose a sentence-level matching framework using open-source LLMs. Extensive baseline experiments on $\textbf{40 MLLMs}$, including 9 reasoning-specific models and 8 reward models, demonstrate that open-source MLLMs still lag significantly behind commercial MLLMs in multi-image reasoning tasks. Furthermore, current multimodal reward models are nearly incapable of handling multi-image reward ranking tasks.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Cheng, Z., Xu, B., Lisheng, G., Zuhe, S., Zhou, T., Shiqi, Z., Ren, S., Chen, M., Meng, X., Zhang, Y., Li, Y., Ren, L., Chen, W., Huang, Z., Zhan, M., Wang, X., & Feng, F. (2025). Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2506.04280

MLA 9

Cheng, Ziming, et al. “Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark.” arXiv (Cornell University), 2025. https://doi.org/10.48550/arxiv.2506.04280.

Chicago

Cheng, Ziming, Bin Xu, Gong, Lisheng, Song, Zuhe, Tianshuo Zhou, Zhong, Shiqi, Siyu Ren, et al. 2025. “Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2506.04280.

Harvard

Cheng, Z. et al. (2025) ‘Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark’, arXiv (Cornell University). doi:10.48550/arxiv.2506.04280.

IEEE

Z. Cheng et al., “Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark,” arXiv (Cornell University), 2025, doi: 10.48550/arxiv.2506.04280.

BibTeX

@misc{cheng2025evaluating,
  title     = {{Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark}},
  author    = {Cheng, Ziming and Xu, Bin and Lisheng, Gong, and Zuhe, Song, and Zhou, Tianshuo and Shiqi, Zhong, and Ren, Siyu and Chen, Mingxiang and Meng, Xiangchao and Zhang, Yuxin and Li, Yanlin and Ren, Lei and Chen, Wei and Huang, Zhiyuan and Zhan, Mingjie and Wang, Xiaojie and Feng, Fangxiang},
  howpublished = {arXiv (Cornell University)},
  year      = {2025},
  doi       = {10.48550/arxiv.2506.04280},
  url       = {https://doi.org/10.48550/arxiv.2506.04280}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications

  • multimodal large language models
  • benchmark
  • spatial reasoning
  • temporal reasoning
  • semantic reasoning

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Cheng, Z., Xu, B., Lisheng, G., Zuhe, S., Zhou, T., Shiqi, Z., Ren, S., Chen, M., Meng, X., Zhang, Y., Li, Y., Ren, L., Chen, W., Huang, Z., Zhan, M., Wang, X., & Feng, F. (2025). Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2506.04280

How many times has this paper been cited?

23 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Ziming Cheng, Bin Xu, Gong, Lisheng, Song, Zuhe, Tianshuo Zhou, Zhong, Shiqi, Siyu Ren, Mingxiang Chen, Xiangchao Meng, Yuxin Zhang, Yanlin Li, Lei Ren and 5 more.

What is the DOI of this paper?

10.48550/arxiv.2506.04280, permanently at https://doi.org/10.48550/arxiv.2506.04280. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.