Uncategorized @en

Speaker: Prof. Panos Kalnis, (KAUST) Google Scholar 2020, Feb. 6, 12pm @ UniPi Conference hall (central building, ground floor) Abstract: Network communication is a major bottleneck in large-scale distributed deep learning. To minimize the problem, many compressed communication schemes, in the form of quantization or sparsification, have been proposed. We investigate them from the Computer Systems perspective, under real-life deployments. We identify discrepancies between the theoretical proposals and the actual implementations, and analyze the impact on convergence. We also develop a general framework for compressed communication that supports both TensorFlow and PyTorch, and employ it to implement many existing schemes. We provide a thorough quantitative evaluation over real models and datasets. We demonstrate the effect of the various overheads, such as the computational cost of compression/decompression, on throughput. Finally, we…