Conference paper · 2025 · Open access
Distilling Multi-Modal Large Language Models for Autonomous Driving
IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 27575–27585 · Published 10 June 2025 · DOI 10.1109/cvpr52734.2025.02568
In brief
“Distilling Multi-Modal Large Language Models for Autonomous Driving” is a 2025 conference paper by Deepti Hegde and 9 co-authors, published in IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings. OpenAlex counts 78 citations as of 21 November 2026, which puts it in the top 1% of works published in 2025 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at arXiv (Cornell University).
- Citations
- 78
- Most cited in
- 2026 (51)
- Field-weighted impact
- 40.15
- References
- 56
- Authors
- 10
Abstract
Autonomous driving demands safe motion planning, especially in critical "long-tail" scenarios. Recent end-to-end autonomous driving systems leverage large language models (LLMs) as planners to improve generalizability to rare events. However, using LLMs at test time introduces high computational costs. To address this, we propose DiMA, an end-to-end autonomous driving system that maintains the efficiency of an LLM-free (or vision-based) planner while leveraging the world knowledge of an LLM. DiMA distills the information from a multi-modal LLM to a vision-based end-to-end planner through a set of specially designed surrogate tasks. Under a joint training strategy, a scene encoder common to both networks produces structured representations that are semantically grounded as well as aligned to the final planning objective. Notably, the LLM is optional at inference, enabling robust planning without compromising on efficiency. Training with DiMA results in a 37% reduction in the L2 trajectory error and an 80% reduction in the collision rate of the vision-based planner, as well as a 44% trajectory error reduction in long-tail scenarios. DiMA also achieves state-of-the-art performance on the nuScenes planning benchmark.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Hegde, D., Yasarla, R., Cai, H., Han, S., Bhattacharyya, A., Mahajan, S., Liu, L., Garrepalli, R., Patel, V. M., & Porikli, F. (2025). Distilling Multi-Modal Large Language Models for Autonomous Driving. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 27575–27585. https://doi.org/10.1109/cvpr52734.2025.02568
MLA 9
Hegde, Deepti, et al. “Distilling Multi-Modal Large Language Models for Autonomous Driving.” IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 2025, pp. 27575–27585. https://doi.org/10.1109/cvpr52734.2025.02568.
Chicago
Hegde, Deepti, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Risheek Garrepalli, Vishal M. Patel, and Fatih Porikli. 2025. “Distilling Multi-Modal Large Language Models for Autonomous Driving.” IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings: 27575–27585. https://doi.org/10.1109/cvpr52734.2025.02568.
Harvard
Hegde, D. et al. (2025) ‘Distilling Multi-Modal Large Language Models for Autonomous Driving’, IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, pp. 27575–27585. doi:10.1109/cvpr52734.2025.02568.
IEEE
D. Hegde et al., “Distilling Multi-Modal Large Language Models for Autonomous Driving,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, pp. 27575–27585, 2025, doi: 10.1109/cvpr52734.2025.02568.
BibTeX
@inproceedings{hegde2025distilling,
title = {{Distilling Multi-Modal Large Language Models for Autonomous Driving}},
author = {Hegde, Deepti and Yasarla, Rajeev and Cai, Hong and Han, Shizhong and Bhattacharyya, Apratim and Mahajan, Shweta and Liu, Litian and Garrepalli, Risheek and Patel, Vishal M. and Porikli, Fatih},
booktitle = {IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings},
year = {2025},
pages = {27575--27585},
doi = {10.1109/cvpr52734.2025.02568},
url = {https://doi.org/10.1109/cvpr52734.2025.02568}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- autonomous driving
- motion planning
- large language models
- knowledge distillation
- end-to-end planning
- long-tail scenarios
- joint training
- inference efficiency
Work with this paper
Most-cited papers in Engineering
- A Closed‐form Equation for Predicting the Hydraulic Conductivity of Unsaturated SoilsMartinus Th. van Genuchten · Soil Science Society of America Journal · 1980 · 28,197 citations
- An improved technique for determining hardness and elastic modulus using load and displacement sensing indentation experimentsW. C. Oliver, George M. Pharr · Journal of materials research/Pratt's guide to venture capital sources · 1992 · 27,379 citations
- Digital communicationsOwen Bishop · Routledge eBooks · 2007 · 24,890 citations
- The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysisNorden Eh Huang, Zhengwei Shen, Steven R. Long and 6 more · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · 1998 · 24,008 citations
- Numerical Heat Transfer and Fluid FlowSuhas V. Patankar · CRC Press eBooks · 2018 · 23,491 citations
Questions about this paper
How do I cite this paper in APA style?
Hegde, D., Yasarla, R., Cai, H., Han, S., Bhattacharyya, A., Mahajan, S., Liu, L., Garrepalli, R., Patel, V. M., & Porikli, F. (2025). Distilling Multi-Modal Large Language Models for Autonomous Driving. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 27575–27585. https://doi.org/10.1109/cvpr52734.2025.02568
How many times has this paper been cited?
78 times, according to OpenAlex on 21 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.
Who wrote this paper?
Deepti Hegde, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Risheek Garrepalli, Vishal M. Patel, Fatih Porikli.
What is the DOI of this paper?
10.1109/cvpr52734.2025.02568, permanently at https://doi.org/10.1109/cvpr52734.2025.02568. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 21 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.