Abstract
Expanding access to practical quantum computers prompts a widespread need to evaluate their performance. Principles and guidelines for carrying out sound empirical work on quantum computing systems are proposed. The guidelines draw heavily on classical experience in experimental algorithmics and computer systems performance analysis, with some adjustments to address issues in quantum computing. The focus is on issues related to quantum annealing processors, although much of the discussion applies to more general scenarios.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
A solver is an algorithm that has been implemented in code or hardware. We assume throughout that the quantum algorithm and platform are tested as a unit.
- 2.
Because over 60% of integers are divisible by primes less than or equal to 11.
- 3.
The No Free Lunch theorem is a formal result that applies to a certain class of algorithms; the No Free Lunch principle is a folklore rule that applies to heuristics. The theorem and the principle apply to classical but not to quantum solvers.
References
Reverse quantum annealing for local refinement of solutions. D-Wave Whitepaper, 14-1018A-A (2017)
Andriyash, E., et al.: Boosting integer factoring performance via quantum annealing offsets. 14-1002A-B (2016)
Bailey, D.H.: Misleading performance reporting in the supercomputing field. Sci. Program. 1(2), 141–151 (1992)
Bailey, D.H.: 12 ways to fool the masses: Fast forward to 2011 (Powerpoint) crd.lbl.gov/~dhbailey (2011). An earlier version appeared as, 1991 “Twelve ways to fool the masses when giving performance results on parallel computers. Supercomputing Review 4(8), 54–55
Barr, R.S., Golden, B.L., Kelly, J.P., Resende, M.G.C., Steward Jr., W.R.: Designing and reporting on computational experiments with heuristic methods. J. Heuristics 1, 9–32 (1995)
Barr, R.S., Hickman, B.L.: Reporting computational experiments with parallel algorithms: issues, measures, and experts’ opinions. ORSA J. Comput. 5(1), 2–18 (1993)
Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds.): Experimental Methods for the Analysis of Optimization Algorithms. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-02538-9
Boixo, S., et al.: Characterizing quantum supremacy in near-term devices. Nat. Phys. 14, 595–600 (2018)
Cohen, P.R.: Empirical Methods for Artificial Intelligence. MIT Press, Cambridge (1995)
Denchev, V.S., et al.: What is the computational value of finite range tunneling? Phys. Rev. X 5, 031026 (2016)
Hen, I., Job, J., Albash, T., Rønnow, T.R., Troyer, M., Lidar, D.: Probing for quantum speedup in spin glass problems with planted solutions. arXiv:1502.01663 (2015)
Hockney, R.W.: The Science of Computer Benchmarking. SIAM, Philadelphia (1996)
Jain, R.: The Art of Computer Systems Performance Analysis. Wiley, Hoboken (1991)
Job, J., Lidar, D.A.: Test-driving 1000 qubits. arXiv:1706.07124 (2017)
Johnson, D.S.: A theoretician’s guide to the experimental analysis of algorithms. In: Goldwasser, M.H., et al. (eds.) Data Structures, Near Neighbor Searches, and Methodology: Fifth and Sixth DIMACS Implementation Challenges. Discrete Mathematics and Theoretical Computer Science, AMS, vol. 59 (2002)
Katzgraber, H.G., Hamze, F., Zhu, Z., Ochoa, A.J., Munoz-Bauza, H.: Seeking quantum speedup through spin glasses: the good, the bad, and the ugly. Phys. Rev. X 5, 031026 (2015)
King, A.D., Lanting, T., Harris, R.: Performance of a quantum annealer on range-limited constraint satisfaction problems. arXiv:1502.02089 (2015)
Macready, W.G., Wolpert, D.H.: What makes an optimization problem hard? Complexity 5, 40–46 (1996)
Mandrà , S., Katzgraber, H.G.: A deceptive step towards quantum speedup detection. arXiv:1711.03168 (2018)
Marshall, J., Venturelli, D., Hen, I., Rieffel, E.G.: The power of pausing: advancing understanding of thermalization in experimental quantum annealers. arXiv:1810.05581 (2016)
McGeoch, C.C.: Toward an experimental method for algorithm simulation (feature article). INFORMS J. Comput. 8(1), 1–15 (1995)
McGeoch, C.C.: A Guide to Experimental Algorithmics. Cambridge Press, Cambridge (2012)
McGeoch, C., Sanders, P., Fleischer, R., Cohen, P.R., Precup, D.: Using finite experiments to study asymptotic performance. In: Fleischer, R., Moret, B., Schmidt, E.M. (eds.) Experimental Algorithmics. LNCS, vol. 2547, pp. 93–126. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-36383-1_5
Panny, W.: Deletions in random binary search trees: a story of errors. J. Stat. Plan. Infer. 140(8), 2335–2345 (2010)
Parekh, O., et al.: Benchmarking Adiabatic Quantum Optimization for Complex Network Analysis, volume SAND2015-3025. Sandia Report, April 2015
Rønnow, T.F., et al.: Defining and detecting quantum speedup. Science 345(6195), 420–424 (2014)
Trummer, I., Koch, C.: Multiple query optimization on the D-Wave 2x adiabatic quantum computer. VLDB 9, 648–659 (2016)
Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1, 67–82 (1997)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
McGeoch, C.C. (2019). Principles and Guidelines for Quantum Performance Analysis. In: Feld, S., Linnhoff-Popien, C. (eds) Quantum Technology and Optimization Problems. QTOP 2019. Lecture Notes in Computer Science(), vol 11413. Springer, Cham. https://doi.org/10.1007/978-3-030-14082-3_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-14082-3_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-14081-6
Online ISBN: 978-3-030-14082-3
eBook Packages: Computer ScienceComputer Science (R0)