Performance Analysis of the NWChem TCE for Different Communication Patterns

  • Priyanka Ghosh
  • Jeff R. HammondEmail author
  • Sayan Ghosh
  • Barbara Chapman
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8551)


One-sided communication is a model that separates communication from synchronization, and has been in practice for over two decades in libraries such as SHMEM and Global Arrays (GA). GA is used in a number of application codes, especially NWChem, and provides a superset of SHMEM functionality that includes remote accumulate, among other features. Remote accumulate is an active-message operation that applies \(y+=a*x\) at the target rather than just \(y=x\) (as in Put) which gives the programmer additional choices with respect to algorithm design. In this paper, we discuss and evaluate communication scenarios for dense block-tensor contractions, one of the mainstays of the NWChem computation chemistry package. We show that apart from the classical approach involving dynamic scheduling of data blocks for load balancing, reordering one-sided Get and Accumulate calls affects the performance of tensor contractions on leadership-class machines substantially. In order to understand why this reordering affects the performance, we develop a proxy application for the NWChem Tensor Contraction Engine (TCE) module. We utilize this proxy application to compare different implementations with a focus on communication.


NWChem One-sided communication Global Arrays MPI-3 Tensor contractions 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aprà, E., Rendell, A.P., Harrison, R.J., Tipparaju, V., de Jong, W.A., Xantheas, S.S.: Liquid water: obtaining the right answer for the right reasons. In: Proceedings of the ACM/IEEE SC Conference on High Performance Networking and Computing, pp. 1–7. ACM, New York (2009)Google Scholar
  2. 2.
    Auer, A.A., Baumgartner, G., Bernholdt, D.E., Bibireata, A., Choppella, V., Cociorva, D., Gao, X., Harrison, R., Krishnamoorthy, S., Krishnan, S., Lam, C.-C., Lu, Q., Nooijen, M., Pitzer, R., Ramanujam, J., Sadayappan, P., Sibiryakov, A.: Automatic code generation for many-body electronic structure methods: the tensor contraction engine. Molecular Physics 104(2), 211–228 (2006)CrossRefGoogle Scholar
  3. 3.
    Bylaska, E.J., de Jong, W.A., Govind, N., Kowalski, K., Straatsma, T.P., Valiev, M., van Dam, H.J.J., Wang, D., Aprà, E., Windus, T.L., Hammond, J., Autschbach, J., Nichols, P., Hirata, S., Hackler, M.T., Zhao, Y., Fan, P.-D., Harrison, R.J., Dupuis, M., Smith, D.M.A., Nieplocha, J., Tipparaju, V., Krishnan, M., Vazquez-Mayagoitia, A., Wu, Q., Voorhis, T.V., Auer, A.A., Nooijen, M., Crosby, L.D., Brown, E., Cisneros, G., Fann, G.I., Früchtl, H., Garza, J., Hirao, K., Kendall, R., Nichols, J.A., Tsemekhman, K., Wolinski, K., Anchell, J., Bernholdt, D., Borowski, P., Clark, T., Clerc, D., Dachsel, H., Deegan, M., Dyall, K., Elwood, D., Glendening, E., Gutowski, M., Hess, A., Jaffe, J., Johnson, B., Ju, J., Kobayashi, R., Kutteh, R., Lin, Z., Littlefield, R., Long, X., Meng, B., Nakajima, T., Niu, S., Pollack, L., Rosing, M., Sandrone, G., Stave, M., Taylor, H., Thomas, G., van Lenthe, J., Wong, A., Zhang, Z.: NWChem, a computational chemistry package for parallel computers, version 6.0 (2010)Google Scholar
  4. 4.
    Goto, K.: Gotoblas. Texas Advanced Computing Center, University of Texas at Austin, USA (2007),
  5. 5.
    Hammond, J.R., Krishnamoorthy, S., Shende, S., Romero, N.A., Malony, A.D.: Performance characterization of global address space applications: a case study with NWChem. Concurrency and Computation: Practice and Experience 24, 135–154 (2011)CrossRefGoogle Scholar
  6. 6.
    Heroux, M.A., Doerfler, D.W., Crozier, P.S., Willenbring, J.M., Edwards, H.C., Williams, A., Rajan, M., Keiter, E.R., Thornquist, H.K., Numrich, R.W.: Improving performance via mini-applications. Sandia National Laboratories, Tech. Rep. SAND2009-5574 (2009)Google Scholar
  7. 7.
    Hirata, S.: Tensor contraction engine: Abstraction and automated parallel implementation of configuration-interaction, coupled-cluster, and many-body perturbation theories. The Journal of Physical Chemistry A 107(46), 9887–9897 (2003)CrossRefGoogle Scholar
  8. 8.
    Kowalski, K., Hammond, J.R., de Jong, W.A., Fan, P.-D., Valiev, M., Wang, D., Govind, N.: Coupled cluster calculations for large molecular and extended systems. In: Reimers, J.R. (ed.) Computational Methods for Large Systems: Electronic Structure Approaches for Biotechnology and Nanotechnology. Wiley (2011)Google Scholar
  9. 9.
    Liu, X., Patel, A., Chow, E.: A new scalable parallel algorithm for fock matrix construction, pp. 1–12 (May 2014)Google Scholar
  10. 10.
    MPI Forum. MPI: A message-passing interface standard. Version 3.0 (November 2012)Google Scholar
  11. 11.
    Nieplocha, J., Harrison, R.J., Littlefield, R.J.: Global arrays: A portable “shared-memory” programming model for distributed memory computers. In: Supercomputing (SC) (1994)Google Scholar
  12. 12.
    Ozog, D., Hammond, J.R., Dinan, J., Balaji, P., Shende, S., Malony, A.: Inspector-executor load balancing algorithms for block-sparse tensor contractions. In: International Conference on Parallel Processing (ICPP) (October 2013)Google Scholar
  13. 13.
    Ozog, D., Shende, S., Malony, A.D., Hammond, J.R., Dinan, J., Balaji, P.: Inspector/executor load balancing algorithms for block-sparse tensor contractions. In: ICS, pp. 483–484 (2013)Google Scholar
  14. 14.
    Rajbhandari, S., Nikam, A., Lai, P.-W., Stock, K., Krishnamoorthy, S., Sadayappan, P.: Framework for distributed contractions of tensors with symmetry. Ohio State University (2013) (Preprint)Google Scholar
  15. 15.
    Solomonik, E., Matthews, D., Hammond, J., Demmel, J.: Cyclops tensor framework: reducing communication and eliminating load imbalance in massively parallel contractions. In: Proceedings of the International Parallel and Distributed Processing Symposium (IPDPS) (2013)Google Scholar
  16. 16.
    Straatsma, T.P., McCammon, J.A.: Load balancing of molecular dynamics simulation with NWChem. IBM Systems Journal 40(2), 328–341 (2001)CrossRefGoogle Scholar
  17. 17.
    Valiev, M., Bylaska, E., Govind, N., Kowalski, K., Straatsma, T., Dam, H.V., Wang, D., Nieplocha, J., Apra, E., Windus, T., de Jong, W.: NWChem: A comprehensive and scalable open-source solution for large scale molecular simulations. Computer Physics Communications 181(9), 1477–1489 (2010)CrossRefzbMATHGoogle Scholar
  18. 18.
    Whaley, R.C., Dongarra, J.J.: Automatically tuned linear algebra software. In: Proceedings of the 1998 ACM/IEEE Conference on Supercomputing (CDROM), pp. 1–27. IEEE Computer Society (1998)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Priyanka Ghosh
    • 1
  • Jeff R. Hammond
    • 2
    Email author
  • Sayan Ghosh
    • 1
  • Barbara Chapman
    • 1
  1. 1.Department of Computer ScienceUniversity of HoustonHoustonUSA
  2. 2.Leadership Computing FacilityArgonne National LaboratoryLemontUSA

Personalised recommendations