Abstract
In this paper we propose and prove that cyclic quorum sets can efficiently manage all-pairs computations and data replication. The quorums are O(N/√P) in size, up to 50% smaller than the dual N/√P array implementations, and significantly smaller than solutions requiring all data. Implementation evaluation demonstrated scalability on real datasets with a 7x speed up on 8 nodes with 1/3rd the memory usage per process.
The all-pairs problem requires all data elements to be paired with all other data elements. These all-pair problems occur in many science fields, which has led to their continued interest. Additionally, as datasets grow in size, new methods like these that can reduce memory footprints and distribute work equally across compute nodes will be demanded.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Hedegaard, R.: Handshake Problem. http://mathworld.wolfram.com/HandshakeProblem.html
Chapman, T., Kalyanaraman, A.: An OpenMP algorithm and implementation for clustering biological graphs. In: Proceedings of the First Workshop on Irregular Applications: Architectures and Algorithm (2011)
Reverter, A., Chan, E.K.: Combining partial correlation and an information theory approach to the reversed engineering of gene co-expression networks. Bioinformatics 24(21), 2491–2497 (2008)
Plimpton, S.: Fast parallel algorithms for short-range molecular dynamics. Journal of Computational Physics 117(1), 1–19 (1995)
Chao, C.-M., Wang, Y.-Z.: A multiple rendezvous multichannel MAC protocol for underwater sensor networks. In: Wireless Communications and Networking Conference (WCNC) (2010)
Luk, W.-S., Wong, T.-T.: Two new quorum based algorithms for distributed mutual exclusion. In: Proceedings of the 17th International Conference on Distributed Computing Systems (1997)
Kumar, V., Agarwal, A.: Multi-dimensional grid quorum consensus for high capacity and availability in a replica control protocol. In: High Performance Architecture and Grid Computing, pp. 67–78 (2011)
Maekawa, M.: An algorithm for mutual exclusion in decentralized systems. ACM Transactions on Computer Systems (TOCS) 3(2), 145–159 (1985)
Colbourn, C.J., Dinitz, J.H.: Handbook of combinatorial designs. CRC press
Koesterke, L., Milfeld, K., Vaughn, M., Stanzione, D., Koltes, J., Weeks, N., Reecy, J.: Optimizing the PCIT algorithm on stampede’s Xeon and Xeon Phi processors for faster discovery of biological networks. In: Proceedings of the Conference on XSEDE: Gateway to Discovery (2013)
Driscoll, M., Georganas, E., Koanantakool, P., Solomonik, E., Yelick, K.: A communication-optimal n-body algorithm for direct interactions. In: 2013 IEEE 27th IPDPS (2013)
Moretti, C., Bui, H., Hollingsworth, K., Rich, B., Flynn, P., Thain, D.: All-pairs: An abstraction for data-intensive computing on campus grids. IEEE Transactions on Parallel and Distributed Systems, 33–46 (2010)
Phillips, P.J., Flynn, P.J., Scruggs, T., Bowyer, K.W., Chang, J., Hoffman, K., Marques, J., Min, J., Worek, W.: Overview of the face recognition grand challenge. In: IEEE Computer Society CVPR 2005 (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer Science+Business Media Singapore
About this paper
Cite this paper
Kleinheksel, C.J., Somani, A.K. (2016). Scaling Distributed All-Pairs Algorithms. In: Kim, K., Joukov, N. (eds) Information Science and Applications (ICISA) 2016. Lecture Notes in Electrical Engineering, vol 376. Springer, Singapore. https://doi.org/10.1007/978-981-10-0557-2_25
Download citation
DOI: https://doi.org/10.1007/978-981-10-0557-2_25
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-0556-5
Online ISBN: 978-981-10-0557-2
eBook Packages: EngineeringEngineering (R0)