Preprint · 2021 · Open access
ResT: An Efficient Transformer for Visual Recognition
arXiv (Cornell University), 34 · Published 28 May 2021 · DOI 10.48550/arxiv.2105.13677
In brief
“ResT: An Efficient Transformer for Visual Recognition” is a 2021 preprint by Qinglong Zhang and Yu-Bin Yang, published in arXiv (Cornell University). OpenAlex counts 272 citations as of 22 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).
- Citations
- 272
- Most cited in
- 2024 (75)
- References
- 38
- Authors
- 2
Abstract
This paper presents an efficient multi-scale vision Transformer, called ResT, that capably served as a general-purpose backbone for image recognition. Unlike existing Transformer methods, which employ standard Transformer blocks to tackle raw images with a fixed resolution, our ResT have several advantages: (1) A memory-efficient multi-head self-attention is built, which compresses the memory by a simple depth-wise convolution, and projects the interaction across the attention-heads dimension while keeping the diversity ability of multi-heads; (2) Position encoding is constructed as spatial attention, which is more flexible and can tackle with input images of arbitrary size without interpolation or fine-tune; (3) Instead of the straightforward tokenization at the beginning of each stage, we design the patch embedding as a stack of overlapping convolution operation with stride on the 2D-reshaped token map. We comprehensively validate ResT on image classification and downstream tasks. Experimental results show that the proposed ResT can outperform the recently state-of-the-art backbones by a large margin, demonstrating the potential of ResT as strong backbones. The code and models will be made publicly available at https://github.com/wofmanaf/ResT.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Zhang, Q., & Yang, Y.-B. (2021). ResT: An Efficient Transformer for Visual Recognition. arXiv (Cornell University), 34. https://doi.org/10.48550/arxiv.2105.13677
MLA 9
Zhang, Qinglong, and Yu-Bin Yang. “ResT: An Efficient Transformer for Visual Recognition.” arXiv (Cornell University), vol. 34, 2021. https://doi.org/10.48550/arxiv.2105.13677.
Chicago
Zhang, Qinglong and Yu-Bin Yang. 2021. “ResT: An Efficient Transformer for Visual Recognition.” arXiv (Cornell University) 34. https://doi.org/10.48550/arxiv.2105.13677.
Harvard
Zhang, Q. and Yang, Y.-B. (2021) ‘ResT: An Efficient Transformer for Visual Recognition’, arXiv (Cornell University), 34. doi:10.48550/arxiv.2105.13677.
IEEE
Q. Zhang and Y.-B. Yang, “ResT: An Efficient Transformer for Visual Recognition,” arXiv (Cornell University), vol. 34, 2021, doi: 10.48550/arxiv.2105.13677.
BibTeX
@misc{zhang2021rest,
title = {{ResT: An Efficient Transformer for Visual Recognition}},
author = {Zhang, Qinglong and Yang, Yu-Bin},
howpublished = {arXiv (Cornell University)},
year = {2021},
volume = {34},
doi = {10.48550/arxiv.2105.13677},
url = {https://doi.org/10.48550/arxiv.2105.13677}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2021 | 11 | |
| 2022 | 39 | |
| 2023 | 73 | |
| 2024 | 75 | |
| 2025 | 47 | |
| 2026 | 27 |
Topic and keywords
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- rest
- vision transformer
- multi-scale vision transformer
- depthwise convolution
- efficient transformers
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Zhang, Q., & Yang, Y.-B. (2021). ResT: An Efficient Transformer for Visual Recognition. arXiv (Cornell University), 34. https://doi.org/10.48550/arxiv.2105.13677
How many times has this paper been cited?
272 times, according to OpenAlex on 22 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.
Who wrote this paper?
Qinglong Zhang, Yu-Bin Yang.
What is the DOI of this paper?
10.48550/arxiv.2105.13677, permanently at https://doi.org/10.48550/arxiv.2105.13677. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 22 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.