Journal article · 2026 · Open access
Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization
International Journal of Computer Vision, 134(12) · Published 11 November 2026 · DOI 10.1007/s11263-026-03124-3
In brief
“Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization” is a 2026 journal article by Zhenglun Kong and 6 co-authors, published in International Journal of Computer Vision. OpenAlex counts 0 citations as of 18 November 2026. It is open access under a CC BY-NC-ND licence: the full text is free to read at International Journal of Computer Vision.
- Citations
- 0
- References
- 56
- Authors
- 7
Abstract
Pre-training cost is a crucial problem in the era of the emerging large models. This is mainly caused by high model computation complexity and the million-scale training datasets, which require extensive time to train from scratch. The substantial training cost associated with recent models poses a significant barrier for individuals and institutions with limited resources. This not only impedes their ability to reach or exceed cutting-edge advancements but also restricts the practical deployment and application of these models on resource-limited devices. Most compression algorithms focus only on efficient fine-tuning or inference from the pre-trained dense models, while time-consuming pre-training is still unavoidable. In contrast, this paper points out that training data is redundant, which is the fundamental reason for the tedious training. To address the issue, this paper aims to introduce sparsity into data and proposes an end-to-end approach for pre-training acceleration by exploiting data redundancy from three sparse perspectives, dubbed Tri-Level ViT++ . Specifically, we leverage a hierarchical reduction scheme by exploring the data sparsity under three levels: the number of training samples in the dataset, the number of patches (tokens) in each sample, and the number of connections between tokens that lie in attention weights. With extensive experiments, we demonstrate that our proposed technique can noticeably accelerate both pre-training and inference for various ViT architectures while maintaining accuracy. Remarkably, under certain ratios, we are able to improve the ViT accuracy rather than compromising it. For example, we can achieve 22.3% speedup with 73.0% (+0.8) Top-1 accuracy on Deit-T, and 22.5% speedup with 80.1% (+0.3) Top-1 accuracy on Deit-S. This proves the existence of data redundancy in ViT.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Kong, Z., Yang, C., Shen, X., Tang, H., Yuan, G., Cheng, M.-M., & Wang, Y. (2026). Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization. International Journal of Computer Vision, 134(12). https://doi.org/10.1007/s11263-026-03124-3
MLA 9
Kong, Zhenglun, et al. “Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization.” International Journal of Computer Vision, vol. 134, no. 12, 2026. https://doi.org/10.1007/s11263-026-03124-3.
Chicago
Kong, Zhenglun, Changdi Yang, Xuan Shen, Hao Tang, Geng Yuan, Ming-Ming Cheng, and Yanzhi Wang. 2026. “Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization.” International Journal of Computer Vision 134 (12). https://doi.org/10.1007/s11263-026-03124-3.
Harvard
Kong, Z. et al. (2026) ‘Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization’, International Journal of Computer Vision, 134(12). doi:10.1007/s11263-026-03124-3.
IEEE
Z. Kong et al., “Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization,” International Journal of Computer Vision, vol. 134, no. 12, 2026, doi: 10.1007/s11263-026-03124-3.
BibTeX
@article{kong2026expediting,
title = {{Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization}},
author = {Kong, Zhenglun and Yang, Changdi and Shen, Xuan and Tang, Hao and Yuan, Geng and Cheng, Ming-Ming and Wang, Yanzhi},
journal = {International Journal of Computer Vision},
year = {2026},
volume = {134},
number = {12},
doi = {10.1007/s11263-026-03124-3},
url = {https://doi.org/10.1007/s11263-026-03124-3}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- vision transformer
- data sparsity
- data redundancy
- hierarchical sparsity
- training cost reduction
- sparse attention
- inference efficiency
- resource-constrained deployment
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Kong, Z., Yang, C., Shen, X., Tang, H., Yuan, G., Cheng, M.-M., & Wang, Y. (2026). Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization. International Journal of Computer Vision, 134(12). https://doi.org/10.1007/s11263-026-03124-3
How many times has this paper been cited?
0 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at International Journal of Computer Vision: sign in and open the PDF from this page.
Who wrote this paper?
Zhenglun Kong, Changdi Yang, Xuan Shen, Hao Tang, Geng Yuan, Ming-Ming Cheng, Yanzhi Wang.
What is the DOI of this paper?
10.1007/s11263-026-03124-3, permanently at https://doi.org/10.1007/s11263-026-03124-3. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.