Journal article · 2026 · Open access
Complementary-Label Learning: A Survey of Methods, an Open-Source Toolkit, and Fair Benchmarking Results
ACM Computing Surveys · Published 3 October 2026 · DOI 10.1145/3847508
In brief
“Complementary-Label Learning: A Survey of Methods, an Open-Source Toolkit, and Fair Benchmarking Results” is a 2026 journal article by Tan-Ha Mai and 6 co-authors, published in ACM Computing Surveys. OpenAlex counts 0 citations as of 16 November 2026. It is open access under a CC BY-NC-ND licence: the full text is free to read at ACM Computing Surveys.
- Citations
- 0
- Field-weighted impact
- 1.08
- References
- 34
- Authors
- 7
Abstract
Complementary-label learning (CLL) is a weakly supervised learning paradigm designed for multiclass classification tasks. In CLL, algorithms are provided with complementary labels, which indicate the classes an instance does not belong to, rather than specifying its true class. With the growing research interest in CLL and the unique challenges it presents, particularly in handling incomplete and indirect supervision, a comprehensive and systematic overview of existing methodologies, results, and research directions has become increasingly essential. In this survey, we provide the first systematic synthesis of recent advances in CLL and offer three major contributions. First, we establish a structured taxonomy that clarifies foundational concepts, distinctive algorithmic strategies, key contributions, and their inherent limitations. Second, to support reproducibility and accelerate future research, we introduce a publicly available open-source CLL toolkit that unifies key algorithm implementations and offers ready-to-use benchmarking pipelines. Third, we conduct an extensive benchmarking study covering a wide spectrum of CLL algorithms, using standardized evaluation protocols across both synthetic, human-annotated, and VLM-annotated complementary-label datasets. This provides the most complete performance comparison currently available. Finally, we discuss current open research challenges and identify promising avenues for future investigation, emphasizing improvements to the practical applicability and robustness of CLL in real-world scenarios. Overall, this survey provides an up-to-date, in-depth resource on the evolving CLL landscape.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Mai, T.-H., Ye, N.-X., Wang, W., Niu, G., Ishida, T., Sugiyama, M., & Lin, H.-T. (2026). Complementary-Label Learning: A Survey of Methods, an Open-Source Toolkit, and Fair Benchmarking Results. ACM Computing Surveys. https://doi.org/10.1145/3847508
MLA 9
Mai, Tan-Ha, et al. “Complementary-Label Learning: A Survey of Methods, an Open-Source Toolkit, and Fair Benchmarking Results.” ACM Computing Surveys, 2026. https://doi.org/10.1145/3847508.
Chicago
Mai, Tan-Ha, Nai-Xuan Ye, Wei Wang, Gang Niu, Takashi Ishida, Masashi Sugiyama, and Hsuan-Tien Lin. 2026. “Complementary-Label Learning: A Survey of Methods, an Open-Source Toolkit, and Fair Benchmarking Results.” ACM Computing Surveys. https://doi.org/10.1145/3847508.
Harvard
Mai, T.-H. et al. (2026) ‘Complementary-Label Learning: A Survey of Methods, an Open-Source Toolkit, and Fair Benchmarking Results’, ACM Computing Surveys. doi:10.1145/3847508.
IEEE
T.-H. Mai et al., “Complementary-Label Learning: A Survey of Methods, an Open-Source Toolkit, and Fair Benchmarking Results,” ACM Computing Surveys, 2026, doi: 10.1145/3847508.
BibTeX
@article{mai2026complementary,
title = {{Complementary-Label Learning: A Survey of Methods, an Open-Source Toolkit, and Fair Benchmarking Results}},
author = {Mai, Tan-Ha and Ye, Nai-Xuan and Wang, Wei and Niu, Gang and Ishida, Takashi and Sugiyama, Masashi and Lin, Hsuan-Tien},
journal = {ACM Computing Surveys},
year = {2026},
doi = {10.1145/3847508},
url = {https://doi.org/10.1145/3847508}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- weakly supervised learning
- multi-class classification
- indirect supervision
- open-source software toolkit
- benchmarking
- vision-language models
- synthetic dataset
- human-annotated dataset
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Mai, T.-H., Ye, N.-X., Wang, W., Niu, G., Ishida, T., Sugiyama, M., & Lin, H.-T. (2026). Complementary-Label Learning: A Survey of Methods, an Open-Source Toolkit, and Fair Benchmarking Results. ACM Computing Surveys. https://doi.org/10.1145/3847508
How many times has this paper been cited?
0 times, according to OpenAlex on 16 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at ACM Computing Surveys: sign in and open the PDF from this page.
Who wrote this paper?
Tan-Ha Mai, Nai-Xuan Ye, Wei Wang, Gang Niu, Takashi Ishida, Masashi Sugiyama, Hsuan-Tien Lin.
What is the DOI of this paper?
10.1145/3847508, permanently at https://doi.org/10.1145/3847508. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 16 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.