Advertisement

The Refutation of Amdahl’s Law and Its Variants

  • Ferenc DévaiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10405)

Abstract

Amdahl’s law, imposing a restriction on the speedup achievable by a multiple number of processors, based on the concept of sequential and parallelizable fractions of computations, has been used to justify, among others, asymmetric chip multiprocessor architectures and concerns of “dark silicon”. This paper demonstrates flaws in Amdahl’s law that (i) in theory no inherently sequential fractions of computations exists (ii) sequential fractions appearing in practice are inherently different from parallelizable fractions and therefore usually have different growth rates and that (iii) the time requirement of sequential fractions can be proportional to the number of processors. However, mathematical analyses are also provided to demonstrate that sequential fractions have negligible effect on speedup if the growth rate of the parallelizable fraction is higher than that of the sequential fraction. Examples from computational geometry are given that Amdahl’s law and its variants fail to represent limits to parallel computation. In particular, Gustafson’s law, claimed to be a refutation of Amdahl’s law by some authors, is shown to contradict established theoretical results. We can conclude that no simple formula or law governing concurrency exists.

Notes

Acknowledgements

The author thanks three anonymous reviewers for their support and constructive criticism that helped to improve the presentation of the paper.

References

  1. 1.
    Akl, S.G., Cosnard, M., Ferreira, A.G.: Data-movement-intensive problems: two folk theorems in parallel computation revisited. Theor. Comput. Sci. 95(2), 323–337 (1992). doi: 10.1016/0304-3975(92)90271-G MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Amdahl, G.M.: Validity of the single processor approach to achieving large scale computing capabilities. In: Proceedings of Spring Joint Computer Conference, pp. 483–485. ACM, New York (1967). doi: 10.1145/1465482.1465560
  3. 3.
    Annavaram, M., Grochowski, E., Shen, J.: Mitigating Amdahl’s law through EPI throttling. SIGARCH Comput. Archit. News 33(2), 298–309 (2005). http://doi.acm.org/10.1145/1080695.1069995 CrossRefGoogle Scholar
  4. 4.
    Atallah, M.J., Callahan, P.B., Goodrich, M.T.: P-complete geometric problems. Int. J. Comput. Geom. Appl. 03(04), 443–462 (1993). doi: 10.1142/S0218195993000282 MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Borkar, S.: Thousand core chips: a technology perspective. In: Proceedings of 44th Annual Design Automation Conference, DAC 2007, pp. 746–749. ACM, New York (2007). http://doi.acm.org/10.1145/1278480.1278667
  6. 6.
    Castanho, C.D., Chen, W., Wada, K., Fujiwara, A.: Parallelizability of some P-complete geometric problems in the EREW-PRAM. In: Wang, J. (ed.) COCOON 2001. LNCS, vol. 2108, pp. 59–63. Springer, Heidelberg (2001). doi: 10.1007/3-540-44679-6_7 CrossRefGoogle Scholar
  7. 7.
    Cook, S.A., Dwork, C.: Bounds on the time for parallel RAM’s to compute simple functions. In: Proceedings of 14th Annual ACM Symposium on Theory of Computing, STOC 1982, pp. 231–233. ACM, New York (1982). doi: 10.1137/0215006
  8. 8.
    Denning, P.J., Lewis, T.G.: Exponential laws of computing growth. Commun. ACM 60(1), 54–65 (2017). http://doi.acm.org/10.1145/2976758 CrossRefGoogle Scholar
  9. 9.
    Dévai, F.: An optimal hidden-surface algorithm and its parallelization. In: Murgante, B., Gervasi, O., Iglesias, A., Taniar, D., Apduhan, B.O. (eds.) ICCSA 2011. LNCS, vol. 6784, pp. 17–29. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-21931-3_2 CrossRefGoogle Scholar
  10. 10.
    Dévai, F.: Gustafson’s law contradicts theory results. (Letter to the Editor). Commun. ACM 60(4), 8–9 (2017). http://doi.acm.org/10.1145/3056859 CrossRefGoogle Scholar
  11. 11.
    Dymond, P.W., Tompa, M.: Speedups of deterministic machines by synchronous parallel machines. J. Comput. Syst. Sci. 30(2), 149–161 (1985). doi: 10.1145/800061.808763 MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Ellen, F., Hendler, D., Shavit, N.: On the inherent sequentiality of concurrent objects. SIAM J. Comput. 41(3), 519–536 (2012). doi: 10.1137/08072646X MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Esmaeilzadeh, H., et al.: Power challenges may end the multicore era. Commun. ACM 56(2), 93–102 (2013). http://doi.acm.org/10.1145/2408776.2408797 CrossRefGoogle Scholar
  14. 14.
    Eyerman, S., Eeckhout, L.: Modeling critical sections in Amdahl’s law and its implications for multicore design. SIGARCH Comput. Archit. News 38(3), 362–370 (2010). http://doi.acm.org/10.1145/1816038.1816011 CrossRefGoogle Scholar
  15. 15.
    Fich, F.E., Meyer auf der Heide, F., Ragde, P., Wigderson, A.: One, two, three ... infinity: lower bounds for parallel computation. In: Proceedings of 17th Annual ACM Symposium on Theory of Computing, STOC 1985, pp. 48–58. ACM, New York (1985). http://doi.acm.org/10.1145/22145.22151
  16. 16.
    Fich, F.E., Hendler, D., Shavit, N.: Linear lower bounds on real-world implementations of concurrent objects. In: Proceedings of 46th Annual IEEE Symposium on Foundations of Computer Science, FOCS 2005, pp. 165–173 (2005). http://dx.doi.org/10.1109/SFCS.2005.47
  17. 17.
    Greenlaw, R., Hoover, H.J., Ruzzo, W.L.: Limits to Parallel Computation: P-Completeness Theory. Oxford University Press Inc., New York (1995)zbMATHGoogle Scholar
  18. 18.
    Gustafson, J.L.: Reevaluating Amdahl’s law. Commun. ACM 31(5), 532–533 (1988). doi: 10.1145/42411.42415 CrossRefGoogle Scholar
  19. 19.
    Hennessy, J.L., Patterson, D.A.: Computer Architecture: A Quantitative Approach, 5th edn. Morgan Kaufmann Publishers Inc., San Francisco (2011)zbMATHGoogle Scholar
  20. 20.
    Herlihy, M., Shavit, N.: The Art of Multiprocessor Programming, Revised Reprint, 1st edn. Morgan Kaufmann Publishers Inc., San Francisco (2012)Google Scholar
  21. 21.
    Hill, M.D., Marty, M.R.: Amdahl’s law in the multicore era. Computer 41(7), 33–38 (2008). doi: 10.1109/MC.2008.209 CrossRefGoogle Scholar
  22. 22.
    Juurlink, B., Meenderinck, C.H.: Amdahl’s law for predicting the future of multicores considered harmful. SIGARCH Comput. Archit. News 40(2), 1–9 (2012). doi: 10.1145/2234336.2234338 CrossRefGoogle Scholar
  23. 23.
    Kuck, D.J.: A survey of parallel machine organization and programming. ACM Comput. Surv. 9(1), 29–59 (1977). http://doi.acm.org/10.1145/356683.356686 MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Kumar, R., Tullsen, D.M., Jouppi, N.P., Ranganathan, P.: Heterogeneous chip multiprocessors. Computer 38(11), 32–38 (2005). doi: 10.1109/MC.2005.379 CrossRefGoogle Scholar
  25. 25.
    Lamport, L.: A new solution of Dijkstra’s concurrent programming problem. Commun. ACM 17(8), 453–455 (1974). http://doi.acm.org/10.1145/361082.361093 MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Luccio, F., Pagli, L.: The p-shovelers problem: (computing with time-varying data). SIGACT News 23, 72–75 (1992). doi: 10.1145/130956.130960 CrossRefGoogle Scholar
  27. 27.
    Mak, L.: Parallelism always helps. SIAM J. Comput. 26(1), 153–172 (1997). doi: 10.1137/S0097539794265402 MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    McKenna, M.: Worst-case optimal hidden-surface removal. ACM Trans. Graph. 6, 19–28 (1987). doi: 10.1145/27625.27627 CrossRefGoogle Scholar
  29. 29.
    Mittal, S.: A survey of techniques for architecting and managing asymmetric multicore processors. ACM Comput. Surv. 48(3), 45:1–45:38 (1987). http://doi.acm.org/10.1145/2856125 Google Scholar
  30. 30.
    Morad, A., Yavits, L., Kvatinsky, S., Ginosar, R.: Resistive GP-SIMD processing-in-memory. ACM Trans. Archit. Code Optim. 12(4), 57:1–57:22 (2016). http://doi.acm.org/10.1145/2845084 CrossRefGoogle Scholar
  31. 31.
    Patterson, D., Hennessy, J.: Computer Organization and Design: The Hardware/Software Interface. The Morgan Kaufmann Series in Computer Architecture and Design, ARM®edn. Elsevier Science, Amsterdam (2016)Google Scholar
  32. 32.
    Patterson, D.A., et al.: A case for redundant arrays of inexpensive disks (RAID). SIGMOD Rec. 17(3), 109–116 (1988). doi: 10.1145/971701.50214 CrossRefGoogle Scholar
  33. 33.
    Paul, J.M., Meyer, B.H.: Amdahl’s law revisited for single chip systems. Int. J. Parallel Program. 35(2), 101–123 (2007). http://dx.doi.org/10.1007/s10766-006-0028-8 CrossRefzbMATHGoogle Scholar
  34. 34.
    Preparata, F.P.: Should Amdahl’s law be repealed? (abstract). In: Staples, J., Eades, P., Katoh, N., Moffat, A. (eds.) ISAAC 1995. LNCS, vol. 1004, p. 311. Springer, Heidelberg (1995). doi: 10.1007/BFb0015436 CrossRefGoogle Scholar
  35. 35.
    Reif, J.H.: Depth-first search is inherently sequential. Inf. Process. Lett. 20(5), 229–234 (1985). doi: 10.1016/0020-0190(85)90024-9 MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Roughgarden, T., Vassilvitskii, S., Wang, J.R.: Shuffles and circuits: (on lower bounds for modern parallel computation). In: Proceedings of 28th ACM Symposium on Parallelism in Algorithms and Architectures, SPAA 2016, pp. 1–12. ACM, New York (2016). http://doi.acm.org/10.1145/2935764.2935799
  37. 37.
    Shavit, N.: Data structures in the multicore age. Commun. ACM 54, 76–84 (2011). http://doi.acm.org/10.1145/1897852.1897873 CrossRefGoogle Scholar
  38. 38.
    Suleman, M.A., Mutlu, O., Qureshi, M.K., Patt, Y.N.: Accelerating critical section execution with asymmetric multi-core architectures. SIGPLAN Not. 44(3), 253–264 (2009). http://doi.acm.org/10.1145/1508284.1508274 CrossRefGoogle Scholar
  39. 39.
    Sun, X.H., Chen, Y.: Reevaluating Amdahl’s law in the multicore era. J. Parallel Distrib. Comput. 70(2), 183–188 (2010). doi: 10.1016/j.jpdc.2009.05.002 CrossRefzbMATHGoogle Scholar
  40. 40.
    Valiant, L.G.: Parallelism in comparison problems. SIAM J. Comput. 4(3), 348–355 (1975). http://dx.doi.org/10.1137/0204030 MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Valiant, L.G.: A bridging model for parallel computation. Commun. ACM 33(8), 103–111 (1990). http://doi.acm.org/10.1145/79173.79181 CrossRefGoogle Scholar
  42. 42.
    Woo, D.H., Lee, H.H.: Extending Amdahl’s law for energy-efficient computing in the many-core era. Computer 41(12), 24–31 (2008). doi: 10.1109/MC.2008.494 CrossRefGoogle Scholar
  43. 43.
    Yavits, L., Morad, A., Ginosar, R.: The effect of communication and synchronization on Amdahl’s law in multicore systems. Parallel Comput. 40(1), 1–16 (2014). http://dx.doi.org/10.1016/j.parco.2013.11.001 MathSciNetCrossRefGoogle Scholar
  44. 44.
    Yavits, L., Morad, A., Ginosar, R.: The effect of temperature on Amdahl law in 3D multicore era. IEEE Trans. Comput. 65(6), 2010–2013 (2016). http://dx.doi.org/10.1109/TC.2015.2458865 MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.London South Bank UniversityLondonUK
  2. 2.Hungarian Academy of SciencesBudapestHungary

Personalised recommendations