Preprint · 2021 · Open access
Vision Xformers: Efficient Attention for Image Classification
arXiv (Cornell University) · Published 5 July 2021 · DOI 10.48550/arxiv.2107.02239
In brief
“Vision Xformers: Efficient Attention for Image Classification” is a 2021 preprint by Pranav Jeevan P and Amit Sethi, published in arXiv (Cornell University). OpenAlex counts 30 citations as of 18 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).
- Citations
- 30
- Most cited in
- 2025 (8)
- References
- 24
- Authors
- 2
Abstract
Although transformers have become the neural architectures of choice for natural language processing, they require orders of magnitude more training data, GPU memory, and computations in order to compete with convolutional neural networks for computer vision. The attention mechanism of transformers scales quadratically with the length of the input sequence, and unrolled images have long sequence lengths. Plus, transformers lack an inductive bias that is appropriate for images. We tested three modifications to vision transformer (ViT) architectures that address these shortcomings. Firstly, we alleviate the quadratic bottleneck by using linear attention mechanisms, called X-formers (such that, X in {Performer, Linformer, Nyströmformer}), thereby creating Vision X-formers (ViXs). This resulted in up to a seven times reduction in the GPU memory requirement. We also compared their performance with FNet and multi-layer perceptron mixers, which further reduced the GPU memory requirement. Secondly, we introduced an inductive bias for images by replacing the initial linear embedding layer by convolutional layers in ViX, which significantly increased classification accuracy without increasing the model size. Thirdly, we replaced the learnable 1D position embeddings in ViT with Rotary Position Embedding (RoPE), which increases the classification accuracy for the same model size. We believe that incorporating such changes can democratize transformers by making them accessible to those with limited data and computing resources.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
P, P. J., & Sethi, A. (2021). Vision Xformers: Efficient Attention for Image Classification. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2107.02239
MLA 9
P, Pranav Jeevan, and Amit Sethi. “Vision Xformers: Efficient Attention for Image Classification.” arXiv (Cornell University), 2021. https://doi.org/10.48550/arxiv.2107.02239.
Chicago
P, Pranav Jeevan and Amit Sethi. 2021. “Vision Xformers: Efficient Attention for Image Classification.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2107.02239.
Harvard
P, P. J. and Sethi, A. (2021) ‘Vision Xformers: Efficient Attention for Image Classification’, arXiv (Cornell University). doi:10.48550/arxiv.2107.02239.
IEEE
P. J. P and A. Sethi, “Vision Xformers: Efficient Attention for Image Classification,” arXiv (Cornell University), 2021, doi: 10.48550/arxiv.2107.02239.
BibTeX
@misc{p2021vision,
title = {{Vision Xformers: Efficient Attention for Image Classification}},
author = {P, Pranav Jeevan and Sethi, Amit},
howpublished = {arXiv (Cornell University)},
year = {2021},
doi = {10.48550/arxiv.2107.02239},
url = {https://doi.org/10.48550/arxiv.2107.02239}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2021 | 1 | |
| 2022 | 6 | |
| 2023 | 6 | |
| 2024 | 7 | |
| 2025 | 8 | |
| 2026 | 2 |
Topic and keywords
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- vision transformer
- linear attention
- image classification
- GPU memory reduction
- inductive bias
- rotary position embedding
- FNET
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
P, P. J., & Sethi, A. (2021). Vision Xformers: Efficient Attention for Image Classification. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2107.02239
How many times has this paper been cited?
30 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.
Who wrote this paper?
Pranav Jeevan P, Amit Sethi.
What is the DOI of this paper?
10.48550/arxiv.2107.02239, permanently at https://doi.org/10.48550/arxiv.2107.02239. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.