Conference paper · 2018 · Open access

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs

Shaohuai Shi, Qiang Wang, Xiaowen Chu

IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), 949–957 · Published 1 August 2018 · DOI 10.1109/dasc/picom/datacom/cyberscitec.2018.000-4

Open PDFView at the publisherCite this paper

In brief

“Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs” is a 2018 conference paper by Shaohuai Shi and 2 co-authors, published in IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech). OpenAlex counts 147 citations as of 21 November 2026, which puts it in the top 2% of works published in 2018 by citations. It is open access: the full text is free to read at arXiv (Cornell University).

Citations
147
Most cited in
2019 (25)
Field-weighted impact
12.83
References
45
Authors
3

Abstract

Deep learning frameworks have been widely deployed on GPU servers for deep learning applications in both academia and industry. In training deep neural networks (DNNs), there are many standard processes or algorithms, such as convolution and stochastic gradient descent (SGD), but the running performance of different frameworks might be different even running the same deep model on the same GPU hardware. In this study, we evaluate the running performance of four state-of-the-art distributed deep learning frameworks (i.e., Caffe-MPI, CNTK, MXNet, and TensorFlow) over single-GPU, multi-GPU, and multi-node environments. We first build performance models of standard processes in training DNNs with SGD, and then we benchmark the running performance of these frameworks with three popular convolutional neural networks (i.e., AlexNet, GoogleNet and ResNet-50), after that, we analyze what factors that result in the performance gap among these four frameworks. Through both analytical and experimental analysis, we identify bottlenecks and overheads which could be further optimized. The main contribution is that the proposed performance models and the analysis provide further optimization directions in both algorithmic design and system configuration.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Shi, S., Wang, Q., & Chu, X. (2018). Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs. IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), 949–957. https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.000-4

MLA 9

Shi, Shaohuai, et al. “Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs.” IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), 2018, pp. 949–957. https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.000-4.

Chicago

Shi, Shaohuai, Qiang Wang, and Xiaowen Chu. 2018. “Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs.” IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech): 949–957. https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.000-4.

Harvard

Shi, S., Wang, Q. and Chu, X. (2018) ‘Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs’, IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), pp. 949–957. doi:10.1109/dasc/picom/datacom/cyberscitec.2018.000-4.

IEEE

S. Shi, Q. Wang, and X. Chu, “Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs,” IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), pp. 949–957, 2018, doi: 10.1109/dasc/picom/datacom/cyberscitec.2018.000-4.

BibTeX

@inproceedings{shi2018performance,
  title     = {{Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs}},
  author    = {Shi, Shaohuai and Wang, Qiang and Chu, Xiaowen},
  booktitle = {IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech)},
  year      = {2018},
  pages     = {949--957},
  doi       = {10.1109/dasc/picom/datacom/cyberscitec.2018.000-4},
  url       = {https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.000-4}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
20181212
20192525
20202020
20212323
20221313
20232020
20241616
202577
20261111

Its field-weighted citation impact is 12.83: it has been cited 12.83 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques

  • GPU performance modeling
  • deep neural network training
  • stochastic gradient descent
  • multi-GPU training
  • CNTK
  • MXNet
  • TensorFlow
  • convolutional neural networks

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Shi, S., Wang, Q., & Chu, X. (2018). Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs. IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), 949–957. https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.000-4

How many times has this paper been cited?

147 times, according to OpenAlex on 21 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Shaohuai Shi, Qiang Wang, Xiaowen Chu.

What is the DOI of this paper?

10.1109/dasc/picom/datacom/cyberscitec.2018.000-4, permanently at https://doi.org/10.1109/dasc/picom/datacom/cyberscitec.2018.000-4. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 21 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.