Preprint · 2022 · Open access
Vision Transformer with Super Token Sampling
arXiv (Cornell University) · Published 21 November 2022 · DOI 10.48550/arxiv.2211.11167
In brief
“Vision Transformer with Super Token Sampling” is a 2022 preprint by Huaibo Huang and 4 co-authors, published in arXiv (Cornell University). OpenAlex counts 108 citations as of 22 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).
- Citations
- 108
- Most cited in
- 2025 (49)
- References
- 51
- Authors
- 5
Abstract
Vision transformer has achieved impressive performance for many vision tasks. However, it may suffer from high redundancy in capturing local features for shallow layers. Local self-attention or early-stage convolutions are thus utilized, which sacrifice the capacity to capture long-range dependency. A challenge then arises: can we access efficient and effective global context modeling at the early stages of a neural network? To address this issue, we draw inspiration from the design of superpixels, which reduces the number of image primitives in subsequent processing, and introduce super tokens into vision transformer. Super tokens attempt to provide a semantically meaningful tessellation of visual content, thus reducing the token number in self-attention as well as preserving global modeling. Specifically, we propose a simple yet strong super token attention (STA) mechanism with three steps: the first samples super tokens from visual tokens via sparse association learning, the second performs self-attention on super tokens, and the last maps them back to the original token space. STA decomposes vanilla global attention into multiplications of a sparse association map and a low-dimensional attention, leading to high efficiency in capturing global dependencies. Based on STA, we develop a hierarchical vision transformer. Extensive experiments demonstrate its strong performance on various vision tasks. In particular, without any extra training data or label, it achieves 86.4% top-1 accuracy on ImageNet-1K with less than 100M parameters. It also achieves 53.9 box AP and 46.8 mask AP on the COCO detection task, and 51.9 mIOU on the ADE20K semantic segmentation task. Code is released at https://github.com/hhb072/STViT.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Huang, H., Zhou, X., Cao, J., He, R., & Tan, T. (2022). Vision Transformer with Super Token Sampling. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2211.11167
MLA 9
Huang, Huaibo, et al. “Vision Transformer with Super Token Sampling.” arXiv (Cornell University), 2022. https://doi.org/10.48550/arxiv.2211.11167.
Chicago
Huang, Huaibo, Xiaoqiang Zhou, Jie Cao, Ran He, and T.N. Tan. 2022. “Vision Transformer with Super Token Sampling.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2211.11167.
Harvard
Huang, H. et al. (2022) ‘Vision Transformer with Super Token Sampling’, arXiv (Cornell University). doi:10.48550/arxiv.2211.11167.
IEEE
H. Huang, X. Zhou, J. Cao, R. He, and T. Tan, “Vision Transformer with Super Token Sampling,” arXiv (Cornell University), 2022, doi: 10.48550/arxiv.2211.11167.
BibTeX
@misc{huang2022vision,
title = {{Vision Transformer with Super Token Sampling}},
author = {Huang, Huaibo and Zhou, Xiaoqiang and Cao, Jie and He, Ran and Tan, T.N.},
howpublished = {arXiv (Cornell University)},
year = {2022},
doi = {10.48550/arxiv.2211.11167},
url = {https://doi.org/10.48550/arxiv.2211.11167}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2020 | 1 | |
| 2023 | 4 | |
| 2024 | 30 | |
| 2025 | 49 | |
| 2026 | 24 |
Topic and keywords
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- vision transformer
- superpixels
- global attention
- long-range dependencies
- hierarchical vision transformer
- self-attention
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Huang, H., Zhou, X., Cao, J., He, R., & Tan, T. (2022). Vision Transformer with Super Token Sampling. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2211.11167
How many times has this paper been cited?
108 times, according to OpenAlex on 22 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.
Who wrote this paper?
Huaibo Huang, Xiaoqiang Zhou, Jie Cao, Ran He, T.N. Tan.
What is the DOI of this paper?
10.48550/arxiv.2211.11167, permanently at https://doi.org/10.48550/arxiv.2211.11167. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 22 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.