Conference paper · 2025 · Open access

Distilling Multi-Modal Large Language Models for Autonomous Driving

Deepti Hegde, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Risheek Garrepalli, Vishal M. Patel, Fatih Porikli

IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 27575–27585 · Published 10 June 2025 · DOI 10.1109/cvpr52734.2025.02568

Open PDFView at the publisherCite this paper

In brief

“Distilling Multi-Modal Large Language Models for Autonomous Driving” is a 2025 conference paper by Deepti Hegde and 9 co-authors, published in IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings. OpenAlex counts 78 citations as of 21 November 2026, which puts it in the top 1% of works published in 2025 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at arXiv (Cornell University).

Citations
78
Most cited in
2026 (51)
Field-weighted impact
40.15
References
56
Authors
10

Abstract

Autonomous driving demands safe motion planning, especially in critical "long-tail" scenarios. Recent end-to-end autonomous driving systems leverage large language models (LLMs) as planners to improve generalizability to rare events. However, using LLMs at test time introduces high computational costs. To address this, we propose DiMA, an end-to-end autonomous driving system that maintains the efficiency of an LLM-free (or vision-based) planner while leveraging the world knowledge of an LLM. DiMA distills the information from a multi-modal LLM to a vision-based end-to-end planner through a set of specially designed surrogate tasks. Under a joint training strategy, a scene encoder common to both networks produces structured representations that are semantically grounded as well as aligned to the final planning objective. Notably, the LLM is optional at inference, enabling robust planning without compromising on efficiency. Training with DiMA results in a 37% reduction in the L2 trajectory error and an 80% reduction in the collision rate of the vision-based planner, as well as a 44% trajectory error reduction in long-tail scenarios. DiMA also achieves state-of-the-art performance on the nuScenes planning benchmark.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Hegde, D., Yasarla, R., Cai, H., Han, S., Bhattacharyya, A., Mahajan, S., Liu, L., Garrepalli, R., Patel, V. M., & Porikli, F. (2025). Distilling Multi-Modal Large Language Models for Autonomous Driving. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 27575–27585. https://doi.org/10.1109/cvpr52734.2025.02568

MLA 9

Hegde, Deepti, et al. “Distilling Multi-Modal Large Language Models for Autonomous Driving.” IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 2025, pp. 27575–27585. https://doi.org/10.1109/cvpr52734.2025.02568.

Chicago

Hegde, Deepti, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Risheek Garrepalli, Vishal M. Patel, and Fatih Porikli. 2025. “Distilling Multi-Modal Large Language Models for Autonomous Driving.” IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings: 27575–27585. https://doi.org/10.1109/cvpr52734.2025.02568.

Harvard

Hegde, D. et al. (2025) ‘Distilling Multi-Modal Large Language Models for Autonomous Driving’, IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, pp. 27575–27585. doi:10.1109/cvpr52734.2025.02568.

IEEE

D. Hegde et al., “Distilling Multi-Modal Large Language Models for Autonomous Driving,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, pp. 27575–27585, 2025, doi: 10.1109/cvpr52734.2025.02568.

BibTeX

@inproceedings{hegde2025distilling,
  title     = {{Distilling Multi-Modal Large Language Models for Autonomous Driving}},
  author    = {Hegde, Deepti and Yasarla, Rajeev and Cai, Hong and Han, Shizhong and Bhattacharyya, Apratim and Mahajan, Shweta and Liu, Litian and Garrepalli, Risheek and Patel, Vishal M. and Porikli, Fatih},
  booktitle = {IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings},
  year      = {2025},
  pages     = {27575--27585},
  doi       = {10.1109/cvpr52734.2025.02568},
  url       = {https://doi.org/10.1109/cvpr52734.2025.02568}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety

  • autonomous driving
  • motion planning
  • large language models
  • knowledge distillation
  • end-to-end planning
  • long-tail scenarios
  • joint training
  • inference efficiency

Work with this paper

Most-cited papers in Engineering

All highly cited Engineering papers

Questions about this paper

How do I cite this paper in APA style?

Hegde, D., Yasarla, R., Cai, H., Han, S., Bhattacharyya, A., Mahajan, S., Liu, L., Garrepalli, R., Patel, V. M., & Porikli, F. (2025). Distilling Multi-Modal Large Language Models for Autonomous Driving. IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, 27575–27585. https://doi.org/10.1109/cvpr52734.2025.02568

How many times has this paper been cited?

78 times, according to OpenAlex on 21 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Deepti Hegde, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Risheek Garrepalli, Vishal M. Patel, Fatih Porikli.

What is the DOI of this paper?

10.1109/cvpr52734.2025.02568, permanently at https://doi.org/10.1109/cvpr52734.2025.02568. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 21 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.