Preprint · 2021 · Open access

ResT: An Efficient Transformer for Visual Recognition

Qinglong Zhang, Yu-Bin Yang

arXiv (Cornell University), 34 · Published 28 May 2021 · DOI 10.48550/arxiv.2105.13677

Open PDFView at the publisherCite this paper

In brief

“ResT: An Efficient Transformer for Visual Recognition” is a 2021 preprint by Qinglong Zhang and Yu-Bin Yang, published in arXiv (Cornell University). OpenAlex counts 272 citations as of 22 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).

Citations
272
Most cited in
2024 (75)
References
38
Authors
2

Abstract

This paper presents an efficient multi-scale vision Transformer, called ResT, that capably served as a general-purpose backbone for image recognition. Unlike existing Transformer methods, which employ standard Transformer blocks to tackle raw images with a fixed resolution, our ResT have several advantages: (1) A memory-efficient multi-head self-attention is built, which compresses the memory by a simple depth-wise convolution, and projects the interaction across the attention-heads dimension while keeping the diversity ability of multi-heads; (2) Position encoding is constructed as spatial attention, which is more flexible and can tackle with input images of arbitrary size without interpolation or fine-tune; (3) Instead of the straightforward tokenization at the beginning of each stage, we design the patch embedding as a stack of overlapping convolution operation with stride on the 2D-reshaped token map. We comprehensively validate ResT on image classification and downstream tasks. Experimental results show that the proposed ResT can outperform the recently state-of-the-art backbones by a large margin, demonstrating the potential of ResT as strong backbones. The code and models will be made publicly available at https://github.com/wofmanaf/ResT.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Zhang, Q., & Yang, Y.-B. (2021). ResT: An Efficient Transformer for Visual Recognition. arXiv (Cornell University), 34. https://doi.org/10.48550/arxiv.2105.13677

MLA 9

Zhang, Qinglong, and Yu-Bin Yang. “ResT: An Efficient Transformer for Visual Recognition.” arXiv (Cornell University), vol. 34, 2021. https://doi.org/10.48550/arxiv.2105.13677.

Chicago

Zhang, Qinglong and Yu-Bin Yang. 2021. “ResT: An Efficient Transformer for Visual Recognition.” arXiv (Cornell University) 34. https://doi.org/10.48550/arxiv.2105.13677.

Harvard

Zhang, Q. and Yang, Y.-B. (2021) ‘ResT: An Efficient Transformer for Visual Recognition’, arXiv (Cornell University), 34. doi:10.48550/arxiv.2105.13677.

IEEE

Q. Zhang and Y.-B. Yang, “ResT: An Efficient Transformer for Visual Recognition,” arXiv (Cornell University), vol. 34, 2021, doi: 10.48550/arxiv.2105.13677.

BibTeX

@misc{zhang2021rest,
  title     = {{ResT: An Efficient Transformer for Visual Recognition}},
  author    = {Zhang, Qinglong and Yang, Yu-Bin},
  howpublished = {arXiv (Cornell University)},
  year      = {2021},
  volume    = {34},
  doi       = {10.48550/arxiv.2105.13677},
  url       = {https://doi.org/10.48550/arxiv.2105.13677}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
20211111
20223939
20237373
20247575
20254747
20262727

Topic and keywords

Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications

  • rest
  • vision transformer
  • multi-scale vision transformer
  • depthwise convolution
  • efficient transformers

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Zhang, Q., & Yang, Y.-B. (2021). ResT: An Efficient Transformer for Visual Recognition. arXiv (Cornell University), 34. https://doi.org/10.48550/arxiv.2105.13677

How many times has this paper been cited?

272 times, according to OpenAlex on 22 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Qinglong Zhang, Yu-Bin Yang.

What is the DOI of this paper?

10.48550/arxiv.2105.13677, permanently at https://doi.org/10.48550/arxiv.2105.13677. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 22 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.