Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Classification of global catastrophic risks connected with artificial intelligence


A classification of the global catastrophic risks of AI is presented, along with a comprehensive list of previously identified risks. This classification allows the identification of several new risks. We show that at each level of AI’s intelligence power, separate types of possible catastrophes dominate. Our classification demonstrates that the field of AI risks is diverse, and includes many scenarios beyond the commonly discussed cases of a paperclip maximizer or robot-caused unemployment. Global catastrophic failure could happen at various levels of AI development, namely, (1) before it starts self-improvement, (2) during its takeoff, when it uses various instruments to escape its initial confinement, or (3) after it successfully takes over the world and starts to implement its goal system, which could be plainly unaligned, or feature-flawed friendliness. AI could also halt at later stages of its development either due to technical glitches or ontological problems. Overall, we identified around several dozen scenarios of AI-driven global catastrophe. The extent of this list illustrates that there is no one simple solution to the problem of AI safety, and that AI safety theory is complex and must be customized for each AI development level.

This is a preview of subscription content, log in to check access.


  1. Alexander S (2016) Ascended economy? Star Slate Codex. Accessed 27 Apr 2018

  2. Anderson M (2017) RethinkX: self-driving electric cars will dominate roads by 2030. In: IEEE Spectrum: technology, engineering and science news. Accessed 17 Jul 2017

  3. (2017) Artificial intelligence startups. Accessed 27 Apr 2018

  4. Armstrong S (2017) Good and safe uses of AI Oracles. ArXiv171105541 Cs

  5. Auerbach D (2014) The Most Terrifying Thought Experiment of All Time. In: Slate. Accessed 27 Apr 2018

  6. Baker BH (2000) The gray matter: the forgotten story of the telephone. Telepress, Kent, WA

  7. Bardi U (2008) The Universal Mining Machine. Accessed 27 Apr 2018

  8. Barrett AM, Baum SD (2017) A model of pathways to artificial superintelligence catastrophe for risk and decision analysis. J Exp Theor Artif Intell 29:397–414

  9. BBC (2017) Cyber-attack: europol says it was unprecedented in scale—BBC News. Accessed 17 Jul 2017

  10. Bender J (2014) Russia may still have an automated nuclear launch system aimed across the northern hemisphere. In: Bus. Insid. Accessed 17 Jul 2017

  11. Blair BG (2011) The logic of accidental nuclear war. Brookings Institution Press, Washington, DC

  12. Boles KS, Kannan K, Gill J et al (2017) Digital-to-biological converter for on-demand production of biologics. Nat Biotechnol 35:672–675 2017

  13. Bostrom N (2002) Existential risks: analyzing human extinction scenarios and related hazards. J Evol Technol 9(1):1–30

  14. Bostrom N (2003a) Astronomical waste: The opportunity cost of delayed technological development. Utilitas 15:308–314

  15. Bostrom N (2003b) Are you living in a computer simulation? Publ Philos Q 53(211):243–255

  16. Bostrom N (2006) What is a singleton. Linguist Philos Investig 5:48–54

  17. Bostrom N (2009) Pascal’s mugging. Analysis 69(3):443–445

  18. Bostrom N (2011) Infinite ethics. Anal Metaphys 9–59

  19. Bostrom N (2014) Superintelligence. Oxford University Press, Oxford

  20. Bradbury RJ (2001) Matrioshka brains. preprint.

  21. Carrigan RA Jr (2006) Do potential SETI signals need to be decontaminated? Acta Astronaut 58:112–117

  22. Chalmers DJ (2002) Does conceivability entail possibility? In: Gendler T, Hawthorne J (eds) Conceivability possibility. Oxford University Press, New York pp 145–200

  23. Chiew KL, Yong KSC, Tan CL (2018) A survey of phishing attacks: their types, vectors and technical approaches. Expert Syst Appl 106:1–20

  24. Christiano P (2016) Prosaic AI alignment. Accessed 27 Apr 2018

  25. Clavero M, García-Berthou E (2005) Invasive species are a leading cause of animal extinctions. Trends Ecol Evol 20:110

  26. Cole DD, Denkenberger D, Griswold M et al (2016) Feeding everyone if industry is disabled. In: Proceedings of the 6th international disaster and risk conference. Davos, Switzerland

  27. Critch A (2017) Toward negotiable reinforcement learning: shifting priorities in Pareto optimal sequential decision-making (arXiv:1701.01302)

  28. Daniel M (2017) S-risks: why they are the worst existential risks, and how to prevent them (EAG Boston 2017). Accessed 27 Apr 2018

  29. Dennett DC (1978) Why you can’t make a computer that feels pain. Synthese 38:415–456

  30. Ellison H (1967) I have no mouth, and i must scream. Galaxy Publishing Corp, New York

  31. Enserink M (2011) Scientists brace for media storm around controversial flu studies. In: Sciencemag. Accessed 27 Apr 2018

  32. Freitas R (2000) Some limits to global ecophagy by biovorous nanoreplicators, with public policy recommendations. Foresight Institute Technical Report

  33. Future of Life Institute (2016) Accidental nuclear war: a timeline of close calls. Accessed 4 Nov 2017

  34. Futureworld (2013) Airplane “crashes” as hacker gets control. In: Futureworld. Accessed 27 Apr 2018

  35. Gildert S (2011) Why “computronium” is really “unobtanium” IO9. Accessed 27 Apr 2018

  36. Goertzel B (2012) Should humanity build a global ai nanny to delay the singularity until it’s better understood? J Conscious Stud 19(1–2):96–111

  37. Grace K, Salvatier J, Dafoe A et al (2017) When will AI exceed human performance? evidence from AI experts. (arXiv:1705.08807 [cs.AI])

  38. Granoff J (2016) Donald trump is an existential threat to America and the world. Time

  39. Gwern (2016) Why tool AIs want to be agent AIs.

  40. Hanson R (2008) Catastrophe, social collapse, and human extinction. In: Bostrom N, Cirkovic MM (eds) Global catastrophic risks. Oxford University Press, Oxford, p 554

  41. Hanson R (2016) The age of Em: work, love, and life when robots rule the earth. Oxford University Press, Oxford

  42. Hines N (2016) Neural implants could let hackers hijack your brain. In: Inverse. Accessed 17 Jul 2017

  43. Hume D (1739) A treatise of human nature. Oxford: Clarendon Press, London, UK

  44. Hutter M (2000) A theory of universal artificial intelligence based on algorithmic complexity. ArXiv Prepr Cs0004001

  45. Jenkins A (2018) Uber may not be to blame for self-driving car death in Arizona. Fortune, New York

  46. Joy B (2000) Why the future doesn’t need us. Wired, San Francisco, CA

  47. Kahn H (1959) On thermonuclear war. Princeton University Press, Princeton

  48. Kardashev NS (1985) On the inevitability and the possible structures of supercivilizations. Reidel Publishing Co., Dordrecht, pp 497–504

  49. Karpathy A (2015) The unreasonable effectiveness of recurrent neural networks. Andrej Karpathy Blog.

  50. Kushner D (2013) The real story of stuxnet. IEEE Spectr 50:48–53

  51. LaVictoire P, Fallenstein B, Yudkowsky E et al (2014) Program equilibrium in the prisoner’s dilemma via Löb’s theorem. MIRI

  52. LaVictorie P (2015) An Introduction to Löb’s Theorem in MIRI Research. MIRI, San Francisco CA.

  53. Lem S (1961) Return from the stars. Houghton Mifflin Harcourt, Boston, US

  54. Lem S (1963) Summa technologiae. Suhrkamp, Berlin, Germany

  55. Lem S (1973) The Invincible: science fiction. Sidgwick & Jackson, London, UK

  56. Lenat DB, Brown JS (1984) Why AM and EURISKO appear to work. Artif Intell 23:269–294

  57. LoPucki LM (2017) Algorithmic ENTITIES. Social Science Research Network, Rochester

  58. Lubin G (2016) Data reveals the 20 most popular TV shows of 2016. Business Insider

  59. Mennen A (2017) Existential risk from AI without an intelligence explosion.

  60. Menzel C (2017) Actualism. In: Zalta EN (ed) The stanford encyclopedia of philosophy, 2014th edn. Metaphysics Research Lab, Stanford University, Stanford

  61. Meuhlhauser L (2014) How big is the field of artificial intelligence? (initial findings). Accessed 27 Apr 2018

  62. Muehlhauser L (2011) Intelligence explosion FAQ. Accessed 27 Apr 2018

  63. Mullin G (2017) What is the Blue Whale suicide challenge, how many deaths has the game been linked to so far and is it in the UK? TheSun

  64. Oberhaus D (2017) Watch ‘Slaughterbots’, a warning about the future of killer bots. In: Motherboard. Accessed 17 Dec 2017

  65. Omohundro S (2008) The basic AI drives. In: Wang P, Goertzel B, Franklin S (eds) Proceedings of the 2008 conference on Artificial General Intelligence 2008: proceedings of the First AGI Conference. IOS Press Amsterdam, The Netherlands

  66. Orwell G (1948) 1984. Houghton Mifflin Harcourt, Boston, US

  67. Pinker S (2011) The better angels of our nature: The decline of violence in history and its causes. Penguin, London

  68. Reason J (2000) Human error: models and management. BMJ 320:768–770

  69. Russell S (2017) 3 principles for creating safer AI. Accessed 27 Apr 2018

  70. Saito T, Angles J (2013) Hikikomori: adolescence without end. Univesity Of Minnesota Press, Minnesota

  71. Sarma GP, Hay NJ (2016) Mammalian value systems. (arXiv:1607.08289 [cs.AI])

  72. Schneier B (2017) Perspective | The next ransomware attack will be worse than WannaCry. Wash, Post

  73. Shakirov V (2016) Review of state-of-the-arts in artificial intelligence with application to AI safety problem. (ArXiv Prepr ArXiv160504232)

  74. Shulman C (2010) Omohundro’s “basic AI drives” and catastrophic risks. Accessed 27 Apr 2018

  75. Shulman C (2011) Arms races and intelligence explosions. Singularity Hypotheses. Springer, New York

  76. Sotala K (2016) Decisive strategic advantage without a hard takeoff. Accessed 27 Apr 2018

  77. Sotala K (2017) Disjunctive AI scenarios: Individual or collective takeoff? Accessed 27 Apr 2018

  78. Sotala K, Yampolskiy R (2014) Responses to catastrophic AGI risk: a survey. Phys Scr 90:018001

  79. Srugatsky N, Strugatsky B (1985) The time wanderers. Richardson & Steirman, New York, US

  80. Strugatsky A, Strugatsky B (1976) The final circle of paradise, Translated by Leonid Renen. DAW, New York

  81. Taylor A (2017) Flying around the world in a solar powered plane—the Atlantic

  82. The Telegraph (2009) Russian spacecraft landed on moon hours before Americans. The telegraph. Accessed 27 Apr 2018

  83. Torres P (2014) Why running simulations may mean the end is near. Accessed 27 Apr 2018

  84. Torres P (2016) Problems with defining an existential risk. IEET. Accessed 27 Apr 2018

  85. Turchin A (2018) The risks connected with possibility of finding alien AI code during SETI. Rev J Br Interplanet Soc. Manuscript,

  86. Turchin A, Denkenberger D (2017) Levels of self-improvement. Manuscript, University of Louisville, TN

  87. Turchin A, Denkenberger D (2018a) Military AI as convergent goal of the self-improving AI. In: Yampolskiy R (ed) Artificial intelligence safety and security. CRC Press, Baca Raton

  88. Turchin A, Denkenberger D (2018b) Could slaughterbots wipe out humanity? Assessment of the global catastrophic risk posed by autonomous weapons. Manuscript

  89. Turchin A, Green B, Denkenberger D (2017) multiple simultaneous pandemics as most dangerous global catastrophic risk connected with bioweapons and synthetic biology. Rev Health Secur

  90. Turing AM (1937) On computable numbers, with an application to the Entscheidungsproblem. Proc Lond Math Soc 2:230–265

  91. Velicovich B (2017) I could kill you with a consumer drone. Defense one, Washington, DC

  92. Watkins J (2016) “Shut up and dance”—“Black mirror” series

  93. Wei D (2013) Outside view(s) and MIRI’s FAI endgame. Accessed 27 Apr 2018

  94. Wootson J (2017) Elon Musk doesn’t think we’re prepared to face humanity’s biggest threat: artificial intelligence. Wash, Post

  95. Yampolskiy R (2014) Utility function security in artificially intelligent agents. J Exp Theor Artif Intell JETAI 373–389.

  96. Yampolskiy R (2015a) Artificial superintelligence: a futuristic approach. CRC Press, Boca Raton

  97. Yampolskiy R (2015b) Taxonomy of pathways to dangerous AI. (ArXiv Prepr ArXiv151103246)

  98. Yampolskiy R, Spellchecker M (2016) artificial intelligence safety and cybersecurity: a timeline of AI failures. (ArXiv Prepr ArXiv161007997)

  99. Yudkowsky E (2001) Creating friendly AI 1.0: the analysis and design of benevolent goal architectures. MIRI, San Francisco, CA, pp 1–282

  100. Yudkowsky E (2002) The AI-Box Experiment. Accessed 27 Apr 2018

  101. Yudkowsky E (2003) HUMOR: friendly AI critical failure table. Accessed 27 Apr 2018

  102. Yudkowsky E (2004) Coherent extrapolated volition. Accessed 27 Apr 2018

  103. Yudkowsky E (2008) Artificial intelligence as a positive and negative factor in global risk, in global catastrophic risks. Oxford University Press, Oxford

  104. Yudkowsky E (2015) From AI to zombies. MIRI, San Francisco, CA

  105. Yudkowsky E (2017) Comment on paper clip maximiser scenario. Accessed 27 Apr 2018

  106. Yudkowsky E, Hanson R (2008) The Hanson-Yudkowsky AI-foom debate. In: MIRI Technical report

  107. Yudkowsky E, Herreshoff M (2013) Tiling agents for self-modifying AI, and the Löbian obstacle. Early Draft MIRI

Download references


We would like to thank Roman Yampolskiy and Seth Baum for their interesting ideas in this article. This article represents views of the authors and does not necessarily represent the views of the Global Catastrophic Risk Institute or the Alliance to Feed the Earth in Disasters. No external sources of funding were used for this work.

Author information

Correspondence to Alexey Turchin.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Turchin, A., Denkenberger, D. Classification of global catastrophic risks connected with artificial intelligence. AI & Soc 35, 147–163 (2020).

Download citation


  • Artificial intelligence
  • Global risks
  • Military drones
  • Superintelligence
  • Existential risk