Journal article · 2026 · Open access

Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization

Zhenglun Kong, Changdi Yang, Xuan Shen, Hao Tang, Geng Yuan, Ming-Ming Cheng, Yanzhi Wang

International Journal of Computer Vision, 134(12) · Published 11 November 2026 · DOI 10.1007/s11263-026-03124-3

Open PDFView at the publisherCite this paper

In brief

“Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization” is a 2026 journal article by Zhenglun Kong and 6 co-authors, published in International Journal of Computer Vision. OpenAlex counts 0 citations as of 18 November 2026. It is open access under a CC BY-NC-ND licence: the full text is free to read at International Journal of Computer Vision.

Citations
0
References
56
Authors
7

Abstract

Pre-training cost is a crucial problem in the era of the emerging large models. This is mainly caused by high model computation complexity and the million-scale training datasets, which require extensive time to train from scratch. The substantial training cost associated with recent models poses a significant barrier for individuals and institutions with limited resources. This not only impedes their ability to reach or exceed cutting-edge advancements but also restricts the practical deployment and application of these models on resource-limited devices. Most compression algorithms focus only on efficient fine-tuning or inference from the pre-trained dense models, while time-consuming pre-training is still unavoidable. In contrast, this paper points out that training data is redundant, which is the fundamental reason for the tedious training. To address the issue, this paper aims to introduce sparsity into data and proposes an end-to-end approach for pre-training acceleration by exploiting data redundancy from three sparse perspectives, dubbed Tri-Level ViT++ . Specifically, we leverage a hierarchical reduction scheme by exploring the data sparsity under three levels: the number of training samples in the dataset, the number of patches (tokens) in each sample, and the number of connections between tokens that lie in attention weights. With extensive experiments, we demonstrate that our proposed technique can noticeably accelerate both pre-training and inference for various ViT architectures while maintaining accuracy. Remarkably, under certain ratios, we are able to improve the ViT accuracy rather than compromising it. For example, we can achieve 22.3% speedup with 73.0% (+0.8) Top-1 accuracy on Deit-T, and 22.5% speedup with 80.1% (+0.3) Top-1 accuracy on Deit-S. This proves the existence of data redundancy in ViT.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Kong, Z., Yang, C., Shen, X., Tang, H., Yuan, G., Cheng, M.-M., & Wang, Y. (2026). Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization. International Journal of Computer Vision, 134(12). https://doi.org/10.1007/s11263-026-03124-3

MLA 9

Kong, Zhenglun, et al. “Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization.” International Journal of Computer Vision, vol. 134, no. 12, 2026. https://doi.org/10.1007/s11263-026-03124-3.

Chicago

Kong, Zhenglun, Changdi Yang, Xuan Shen, Hao Tang, Geng Yuan, Ming-Ming Cheng, and Yanzhi Wang. 2026. “Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization.” International Journal of Computer Vision 134 (12). https://doi.org/10.1007/s11263-026-03124-3.

Harvard

Kong, Z. et al. (2026) ‘Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization’, International Journal of Computer Vision, 134(12). doi:10.1007/s11263-026-03124-3.

IEEE

Z. Kong et al., “Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization,” International Journal of Computer Vision, vol. 134, no. 12, 2026, doi: 10.1007/s11263-026-03124-3.

BibTeX

@article{kong2026expediting,
  title     = {{Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization}},
  author    = {Kong, Zhenglun and Yang, Changdi and Shen, Xuan and Tang, Hao and Yuan, Geng and Cheng, Ming-Ming and Wang, Yanzhi},
  journal   = {International Journal of Computer Vision},
  year      = {2026},
  volume    = {134},
  number    = {12},
  doi       = {10.1007/s11263-026-03124-3},
  url       = {https://doi.org/10.1007/s11263-026-03124-3}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications

  • vision transformer
  • data sparsity
  • data redundancy
  • hierarchical sparsity
  • training cost reduction
  • sparse attention
  • inference efficiency
  • resource-constrained deployment

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Kong, Z., Yang, C., Shen, X., Tang, H., Yuan, G., Cheng, M.-M., & Wang, Y. (2026). Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization. International Journal of Computer Vision, 134(12). https://doi.org/10.1007/s11263-026-03124-3

How many times has this paper been cited?

0 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at International Journal of Computer Vision: sign in and open the PDF from this page.

Who wrote this paper?

Zhenglun Kong, Changdi Yang, Xuan Shen, Hao Tang, Geng Yuan, Ming-Ming Cheng, Yanzhi Wang.

What is the DOI of this paper?

10.1007/s11263-026-03124-3, permanently at https://doi.org/10.1007/s11263-026-03124-3. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.