Conference paper · 2018 · Open access
Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), 949–957 · Published 1 August 2018 · DOI 10.1109/dasc/picom/datacom/cyberscitec.2018.000-4
In brief
“Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs” is a 2018 conference paper by Shaohuai Shi and 2 co-authors, published in IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech). OpenAlex counts 147 citations as of 21 November 2026, which puts it in the top 2% of works published in 2018 by citations. It is open access: the full text is free to read at arXiv (Cornell University).
- Citations
- 147
- Most cited in
- 2019 (25)
- Field-weighted impact
- 12.83
- References
- 45
- Authors
- 3
Abstract
Deep learning frameworks have been widely deployed on GPU servers for deep learning applications in both academia and industry. In training deep neural networks (DNNs), there are many standard processes or algorithms, such as convolution and stochastic gradient descent (SGD), but the running performance of different frameworks might be different even running the same deep model on the same GPU hardware. In this study, we evaluate the running performance of four state-of-the-art distributed deep learning frameworks (i.e., Caffe-MPI, CNTK, MXNet, and TensorFlow) over single-GPU, multi-GPU, and multi-node environments. We first build performance models of standard processes in training DNNs with SGD, and then we benchmark the running performance of these frameworks with three popular convolutional neural networks (i.e., AlexNet, GoogleNet and ResNet-50), after that, we analyze what factors that result in the performance gap among these four frameworks. Through both analytical and experimental analysis, we identify bottlenecks and overheads which could be further optimized. The main contribution is that the proposed performance models and the analysis provide further optimization directions in both algorithmic design and system configuration.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Shi, S., Wang, Q., & Chu, X. (2018). Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs. IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), 949–957. https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.000-4
MLA 9
Shi, Shaohuai, et al. “Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs.” IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), 2018, pp. 949–957. https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.000-4.
Chicago
Shi, Shaohuai, Qiang Wang, and Xiaowen Chu. 2018. “Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs.” IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech): 949–957. https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.000-4.
Harvard
Shi, S., Wang, Q. and Chu, X. (2018) ‘Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs’, IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), pp. 949–957. doi:10.1109/dasc/picom/datacom/cyberscitec.2018.000-4.
IEEE
S. Shi, Q. Wang, and X. Chu, “Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs,” IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), pp. 949–957, 2018, doi: 10.1109/dasc/picom/datacom/cyberscitec.2018.000-4.
BibTeX
@inproceedings{shi2018performance,
title = {{Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs}},
author = {Shi, Shaohuai and Wang, Qiang and Chu, Xiaowen},
booktitle = {IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech)},
year = {2018},
pages = {949--957},
doi = {10.1109/dasc/picom/datacom/cyberscitec.2018.000-4},
url = {https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.000-4}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2018 | 12 | |
| 2019 | 25 | |
| 2020 | 20 | |
| 2021 | 23 | |
| 2022 | 13 | |
| 2023 | 20 | |
| 2024 | 16 | |
| 2025 | 7 | |
| 2026 | 11 |
Its field-weighted citation impact is 12.83: it has been cited 12.83 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- GPU performance modeling
- deep neural network training
- stochastic gradient descent
- multi-GPU training
- CNTK
- MXNet
- TensorFlow
- convolutional neural networks
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Shi, S., Wang, Q., & Chu, X. (2018). Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs. IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), 949–957. https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.000-4
How many times has this paper been cited?
147 times, according to OpenAlex on 21 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.
Who wrote this paper?
Shaohuai Shi, Qiang Wang, Xiaowen Chu.
What is the DOI of this paper?
10.1109/dasc/picom/datacom/cyberscitec.2018.000-4, permanently at https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.000-4. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 21 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.