Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models

Citation

If you find this work useful for your research, please cite our paper:

@article{li2025perception,
  title={Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models},
  author={Li, Yunxin and Liu, Zhenyu and Li, Zitao and Zhang, Xuanyu and Xu, Zhenran and Chen, Xinyu and Shi, Haoyuan and Jiang, Shenyuan and Wang, Xintong and Wang, Jifang and Huang, Shouzheng and Zhao, Xinping and Jiang, Borui and Hong, Lanqing and Wang, Longyue and Tian, Zhuotao and Huai, Baoxing and Luo, Wenhan and Luo, Weihua and Zhang, Zheng and Hu, Baotian and Zhang, Min},
  journal={arXiv preprint arXiv:2505.04921},
  year={2025}
}

News

🔥 Latest :point_right: [2025/08/02] We updated the recommended works about multimodal reasoning in our repo and paper. You are welcome to recommend your work to us.

🔥 Latest :point_right: [2025/07/06] We collected recent multimodal reasoning models and benchmarks (about 150 papers in 2025.05~2025.06) in our new version and repository. You are welcome to recommend your work to us.

About

✨ Advances on multimodal reasoning models and a collection of related datasets and benchmarks ✨

Figure 1: The core evolving path of large multimodal reasoning models.

1 Overview

Figure 2: The roadmap of large multimodal reasoning models.

Large Multimodal Reasoning Models (LMRMs) have emerged as a promising paradigm, integrating modalities such as text, images, audio, and video to support complex reasoning capabilities—aiming to achieve comprehensive perception, precise understanding, and deep reasoning.

As research advances, multimodal reasoning has rapidly evolved from modular, perception-driven pipelines to unified, language-centric frameworks that offer more coherent cross-modal understanding. While instruction tuning and reinforcement learning have improved model reasoning, significant challenges remain in omni-modal generalization, reasoning depth, and agentic behavior.

We present a comprehensive and structured survey of multimodal reasoning research, organized around a four-stage developmental roadmap that reflects the field’s shifting design philosophies and emerging capabilities.

First, we review early efforts based on task-specific modules, where reasoning was implicitly embedded across stages of representation, alignment, and fusion.

Next, we examine recent approaches that unify reasoning into multimodal LLMs, with advances such as Multimodal Chain-of-Thought (MCoT) and multimodal reinforcement learning enabling richer and more structured reasoning chains.

Finally, drawing on empirical insights from challenging benchmarks and experimental cases of OpenAI-O3 and O4-mini, we discuss the conceptual direction of native large multimodal reasoning models (N-LMRMs), which aim to support scalable, agentic, and adaptive reasoning and planning in complex, real-world environments.

2 Roadmap of Multimodal Reasoning Models

2.1 Stage 1 Perception Driven Reasoning - Developing Task-Specific Reasoning Modules

2.1.1 Modular Reasoning Networks

2.1.2 Vision-Language Models-based Modular Reasoning

2.2 Stage 2 Language-Centric Short Reasoning - System-1 Reasoning

With the advent of large-scale multimodal pretraining, MLLMs have started to demonstrate emergent reason- ing capabilities. However, such inferences are often shallow, relying primarily on implicit correlations rather than explicit logical processes. To mitigate this limitation, MCoT has emerged as a simple yet effective ap- proach. By incorporating intermediate reasoning steps, MCoT improves cross-modal alignment, knowledge integration, and contextual grounding, all without the need for extensive supervision or significant architec- tural modifications. In this stage, we categorize existing approaches into three paradigms: prompt-based MCoT, structural reasoning with predefined patterns, and tool-augmented reasoning leveraging lightweight external modules.

Figure 3: Taxonomy and representative methods of structural reasoning in multimodal chain-of-thought.

2.2.1 Prompt-based MCoT

2.2.2 Structural Reasoning

2.2.3 Externally Augmented Reasoning

2.3 Stage 3 Language-Centric Long Reasoning - System-2 Thinking and Planning

While structural reasoning introduces predefined patterns to guide MLLMs toward more systematic reason- ing, it remains constrained by shallow reasoning depth and limited adaptability. To handle more complex multimodal tasks, recent work aims to develop System-2-style reasoning (Kahneman, 2011). Unlike fast and reactive strategies, this form of reasoning is deliberate, compositional, and guided by explicit planning. By extending reasoning chains, grounding them in multimodal inputs, and training with supervised or reinforce- ment signals, these models begin to exhibit long-horizon reasoning and adaptive problem decomposition.

Figure4: Timeline (top) and core components (bottom) of recent multimodal O1-like and R1-like models.

2.3.1 Cross-Modal Reasoning

2.3.2 MM-O1

2.3.3 MM-R1

3 Towards Native Multimodal Reasoning Model

Large Multimodal Reasoning Models (LMRMs) have demonstrated potential in handling complex tasks with long chain-of-thought. However, their language-centric architectures constrain their effectiveness in real- world scenarios. Specifically, their reliance on vision and language modalities limits their capacity to process and reason over interleaved diverse data types, while their performance in real-time, iterative interactions with dynamic environments remains underdeveloped. These limitations underscore the need for a new class of models capable of broader multimodal integration and more advanced interactive reasoning.

3.1 Evaluation of O3 and O4-mini

Figure 5: Case study of OpenAI o3’s long multimodal chain-of-thought, reaching the correct answer after 8 minutes and 13 seconds of reasoning.

Figure 6: Case study of OpenAI o3: Find locations, solve a puzzle and create multimedia contents.

Figure 7: Case study of OpenAI o3: Visual problem solving and file processing.

3.2 Model Capability

3.3 Technical Prospect

Figure 8: Overview of next-generation native large multimodal reasoning model. The envisioned system aims to achieve comprehensive perception across diverse real-world data modalities, enabling precise omnimodal understanding and in-depth generative reasoning. This foundational model will lead to more advanced forms of intelligent behavior, learning from world experience and realizing lifelong learning and self-improvement.

4 Dataset and Benchmark

Figure 9: The outlines of datasets and benchmarks. We reorganize the multimodal datasets and benchmarks into four main categories: Understanding, Generation, Reasoning, and Planning.

5 Conclusion

In this paper, we survey the evolution of multimodal reasoning models, highlighting pivotal advancements and paradigm-shifting milestones in the field. While current models predominantly adopt a language-centric reasoning paradigm—delivering impressive results in tasks like visual question answering and text-image retrieval—critical challenges persist. Notably, visual-centric long reasoning (e.g., understanding object relations or 3D contexts, addressing visual information seeking questions) and interactive multimodal reasoning (e.g., dynamic cross-modal dialogue or iterative feedback loops) remain underdeveloped frontiers requiring deeper exploration.

Building on empirical evaluations and experimental insights, we propose a forward-looking framework for inherently multimodal large models that transcend language-dominated architectures. Such models should prioritize three core capabilities:

1. Multimodal Agentic Reasoning: Enabling proactive environmental interaction (e.g., embodied AI agents that learn through real-world trial and error)
2. Omni-Modal Understanding and Generative Reasoning:
   • Integrating any-modal semantics (e.g., aligning abstract concepts across vision, audio, and text) while resolving ambiguities in complex, open-world contexts
   • Producing coherent, context-aware outputs across modalities (e.g., generating diagrams from spoken instructions or synthesizing video narratives from text)

By addressing these dimensions, future models could achieve human-like contextual adaptability, bridging the gap between isolated task performance and generalized, real-world problem-solving.

Acknowledge

We express sincere gratitude for the valuable contributions of all researchers and students involved in this work.

We welcome the community to contribute to the development of this survey, and we will regularly update it to reflect the latest research.

Please feel free to submit issues or contact us via email at liyunxin987@163.com.

Github Star