Journal article · 2026 · Open access

Distilling Transformers: A Survey

Jianping Gou, Liyuan Sun, Baosheng Yu, Philip H. S. Torr, Ming-Hsuan Yang, Dacheng Tao

IEEE Transactions on Pattern Analysis and Machine Intelligence, PP, 1–20 · Published 1 January 2026 · DOI 10.1109/tpami.2026.3751310

Open full textCite this paper

In brief

“Distilling Transformers: A Survey” is a 2026 journal article by Jianping Gou and 5 co-authors, published in IEEE Transactions on Pattern Analysis and Machine Intelligence. OpenAlex counts 0 citations as of 16 November 2026. It is open access under a CC BY licence: the full text is free to read at IEEE Transactions on Pattern Analysis and Machine Intelligence.

Citations
0
Authors
6

Abstract

Transformers have gained widespread recognition across sectors and academic fields for their innovative self-attention mechanism, which effectively captures long-range dependencies, and for their scalability and exceptional performance across diverse applications. However, their high computational cost and time requirements limit their adoption in resource-constrained environments. Knowledge distillation provides a straightforward yet effective approach for compressing traditional network architectures, such as CNNs, by transferring knowledge from a large teacher model to a smaller student model. This paper explores the potential for model compression by combining novel Transformer architectures with knowledge distillation. Specifically, we categorize, summarize, and present the substantial body of work on Transformer knowledge and distillation schemes. Furthermore, this survey extends its coverage to widely used knowledge-distillation pipelines for large language models (LLMs) and systematically analyses six major distillation targets. We also review popular applications in various domains, including natural language processing, computer vision, and speech recognition. Finally, we provide forward-looking perspectives, highlighting potential directions for future research on developing highly efficient yet performance-competitive Transformer models through knowledge distillation techniques.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Gou, J., Sun, L., Yu, B., Torr, P. H. S., Yang, M.-H., & Tao, D. (2026). Distilling Transformers: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP, 1–20. https://doi.org/10.1109/tpami.2026.3751310

MLA 9

Gou, Jianping, et al. “Distilling Transformers: A Survey.” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PP, 2026, pp. 1–20. https://doi.org/10.1109/tpami.2026.3751310.

Chicago

Gou, Jianping, Liyuan Sun, Baosheng Yu, Philip H. S. Torr, Ming-Hsuan Yang, and Dacheng Tao. 2026. “Distilling Transformers: A Survey.” IEEE Transactions on Pattern Analysis and Machine Intelligence PP: 1–20. https://doi.org/10.1109/tpami.2026.3751310.

Harvard

Gou, J. et al. (2026) ‘Distilling Transformers: A Survey’, IEEE Transactions on Pattern Analysis and Machine Intelligence, PP, pp. 1–20. doi:10.1109/tpami.2026.3751310.

IEEE

J. Gou, L. Sun, B. Yu, P. H. S. Torr, M.-H. Yang, and D. Tao, “Distilling Transformers: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PP, pp. 1–20, 2026, doi: 10.1109/tpami.2026.3751310.

BibTeX

@article{gou2026distilling,
  title     = {{Distilling Transformers: A Survey}},
  author    = {Gou, Jianping and Sun, Liyuan and Yu, Baosheng and Torr, Philip H. S. and Yang, Ming-Hsuan and Tao, Dacheng},
  journal   = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year      = {2026},
  volume    = {PP},
  pages     = {1--20},
  doi       = {10.1109/tpami.2026.3751310},
  url       = {https://doi.org/10.1109/tpami.2026.3751310}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications

  • knowledge distillation
  • transformer models
  • model compression
  • self-attention
  • large language models
  • efficient transformers
  • resource-constrained environments
  • natural language processing

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Gou, J., Sun, L., Yu, B., Torr, P. H. S., Yang, M.-H., & Tao, D. (2026). Distilling Transformers: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP, 1–20. https://doi.org/10.1109/tpami.2026.3751310

How many times has this paper been cited?

0 times, according to OpenAlex on 16 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at IEEE Transactions on Pattern Analysis and Machine Intelligence: sign in and open the full text from this page.

Who wrote this paper?

Jianping Gou, Liyuan Sun, Baosheng Yu, Philip H. S. Torr, Ming-Hsuan Yang, Dacheng Tao.

What is the DOI of this paper?

10.1109/tpami.2026.3751310, permanently at https://doi.org/10.1109/tpami.2026.3751310. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 16 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.