Learning How to Behave

Moral Competence for Social Robots
  • Bertram F. MalleEmail author
  • Matthias Scheutz
Living reference work entry
Part of the Springer Reference Geisteswissenschaften book series (SPREFGEIST)


We describe a theoretical framework and recent research on one key aspect of robot ethics: the development and implementation of a robot’s moral competence. As autonomous machines take on increasingly social roles in human communities, these machines need to have some level of moral competence to ensure safety, acceptance, and justified trust. We review the extensive and complex elements of human moral competence and ask how analogous competences could be implemented in a robot. We propose that moral competence consists of five elements, two constituents (moral norms and moral vocabulary) and three activities (moral judgment, moral action, and moral communication). A robot’s computational representations of social and moral norms is a prerequisite for all three moral activities. However, merely programming in advance the vast network of human norms is impossible, so new computational learning algorithms are needed that allow robots to acquire and update the context-specific and graded norms relevant to their domain of deployment. Moral vocabulary is needed primarily for moral communication, which expresses moral judgments of others’ violations and explains one’s own moral violations – to justify them, apologize, or declare intentions to do better. Current robots have at best rudimentary moral competence, but with improved learning and reasoning they may begin to show the kinds of capacities that humans will expect of future social robots.


Norms Robot ethics Machine morality Moral action Moral judgment Machine learning Explanations 



This project was supported by a grant from the Office of Naval Research (ONR), No. N00014-13-1-0269, and from the Defense Advanced Research Projects Agency (DARPA), SIMPLEX 14-46-FP-097. The opinions expressed here are our own and do not necessarily reflect the views of ONR or DARPA.


  1. Aarts, Henk, and Ap Dijksterhuis. 2003. The silence of the library: Environment, situational norm, and social behavior. Journal of Personality and Social Psychology 84(1): 18–28. Scholar
  2. Ågotnes, Thomas, and Michael Wooldridge. 2010. Optimal social laws. In Proceedings of the 9th international conference on autonomous agents and multiagent systems (AAMAS 2010), ed. van der Hoek, Kaminka, Lesprance, Luck, and Sen, 667–674.Google Scholar
  3. Alexander, Richard D. 1987. The biology of moral systems. Hawthorne: Aldine de Gruyter.Google Scholar
  4. Alicke, Mark D. 2000. Culpable control and the psychology of blame. Psychological Bulletin 126(4): 556–574. Scholar
  5. Anderson, Michael, and Susan Leigh Anderson, eds. 2011. Machine ethics. New York: Cambridge University Press.Google Scholar
  6. Andrighetto, Giulia, Guido Governatori, Pablo Noriega, and Leendert W. N van der Torre, eds. 2013. Normative multi-agent systems, Dagstuhl Follow-Ups 4. Saarbrücken/Wadern: Dagstuhl Publishing.
  7. Antaki, Charles. 1994. Explaining and arguing: The social organization of accounts. London: Sage.Google Scholar
  8. Arkin, Ronald C. 2009. Governing lethal behavior in autonomous robots. Boca Raton: CRC Press.CrossRefGoogle Scholar
  9. Arkin, Ronald C., and T. Balch. 1997. AuRA: Principles and practice in review. Journal of Experimental and Theoretical Artificial Intelligence 9(2): 175–189.CrossRefGoogle Scholar
  10. Arkin, Ronald C., and P. Ulam. 2009. An Ethical adaptor: Behavioral modification derived from moral emotions. In Computational intelligence in robotics and automation (CIRA), 2009 IEEE international symposium on, 381–387. Piscataway: IEEE Press.Google Scholar
  11. Baldwin, Dare A., and Michael Tomasello. 1998. Word learning: A window on early pragmatic understanding. In The proceedings of the twenty-ninth annual child language research forum, ed. Eve V. Clark, 3–23. Chicago: Center for the Study of Language and Information.Google Scholar
  12. Bandura, Albert. 1999. Moral disengagement in the perpetration of inhumanities. Personality and Social Psychology Review 3(3): 193–209. Scholar
  13. Bartneck, Christoph, Marcel Verbunt, Omar Mubin, and Abdullah Al Mahmud. 2007. To kill a mockingbird robot. In Proceedings of the ACM/IEEE international conference on human-robot interaction, 81–87. New York: ACM Press.
  14. Bello, Paul. 2012. Cognitive foundations for a computational theory of mindreading. Advances in Cognitive Systems 1: 59–72.Google Scholar
  15. Bicchieri, Cristina. 2006. The grammar of society: The nature and dynamics of social norms. New York: Cambridge University Press.Google Scholar
  16. Blass, Joseph A., and Kenneth D. Forbus. 2015. Moral decision-making by analogy: Generalizations versus exemplars. In Twenty-Ninth AAAI conference on artificial intelligence, 501–507. AAAI.
  17. Bloom, Paul. 2016. Against empathy: The case for rational compassion. New York: Ecco.Google Scholar
  18. Bostrom, Nick. 2014. Superintelligence: Paths, dangers, strategies. Oxford: Oxford University Press.Google Scholar
  19. Brennan, Geoffrey, Lina Eriksson, Robert E. Goodin, and Nicholas Southwood. 2013. Explaining norms. New York: Oxford University Press.CrossRefGoogle Scholar
  20. Briggs, Gordon, and Matthias Scheutz. 2013. A hybrid architectural approach to understanding and appropriately generating indirect speech acts. In Proceedings of twenty-seventh AAAI conference on artificial intelligence, 1213–1219.Google Scholar
  21. Briggs, Gordon, and Matthias Scheutz. 2015. “‘Sorry, I can’t do that:’ Developing mechanisms to appropriately reject directives in human-robot interactions.” In Proceedings of the 2015 AAAI fall symposium on AI and HRI, 32–36. Palo Alto, CA: AAAI Press.Google Scholar
  22. Bringsjord, Selmer. 2015. A vindication of program verification. History and Philosophy of Logic 36(3): 262–277. Scholar
  23. Bringsjord, Selmer, Konstantine Arkoudas, and Paul Bello. 2006. Toward a general logicist methodology for engineering ethically correct robots. Intelligent Systems, IEEE 21(4): 38–44.CrossRefGoogle Scholar
  24. Broersen, Jan, Mehdi Dastani, Joris Hulstijn, Zisheng Huang, and Leendert van der Torre. 2001. The BOID architecture: Conflicts between beliefs, obligations, intentions and desires. In Proceedings of the fifth international conference on autonomous agents, AGENTS ’01, 9–16. New York: ACM.
  25. Casasanto, Daniel, and Lera Boroditsky. 2008. Time in the mind: Using space to think about time. Cognition 106(2): 579–593. Scholar
  26. Casler, Krista, Treysi Terziyan, and Kimberly Greene. 2009. Toddlers view artifact function normatively. Cognitive Development 24(3): 240–247. Scholar
  27. Chappie. 2015. Motion picture. Sony Pictures Home Entertainment.Google Scholar
  28. Churchland, Patricia S. 2012. Braintrust: What neuroscience tells us about morality. Princeton: Princeton University Press.Google Scholar
  29. Cialdini, Robert B., Raymond R. Reno, and Carl A. Kallgren. 1990. A focus theory of normative conduct: Recycling the concept of norms to reduce littering in public places. Journal of Personality and Social Psychology 58(6): 1015–1026. Scholar
  30. Cima, Maaike, Franca Tonnaer, and Marc D. Hauser. 2010. Psychopaths know right from wrong but don’t care. Social Cognitive and Affective Neuroscience 5(1): 59–67. Scholar
  31. Clark, Herbert H. 1985. Language use and language users. In Handbook of social psychology, ed. Gardner Lindzey and Eliot Aronson, 179–231. New York: Random House.Google Scholar
  32. Coeckelbergh, Mark. 2010. Robot rights? Towards a social-relational justification of moral consideration. Ethics and Information Technology 12(3): 209–221. Scholar
  33. Conte, Rosaria, Giulia Andrighetto, and Marco Campenni. 2013. Minding norms: Mechanisms and dynamics of social order in agent societies. New York: Oxford University Press.CrossRefGoogle Scholar
  34. Cox, Michael T. 2011. Metareasoning, monitoring, and self-explanation. In Metareasoning, ed. Michael T. Cox and Anita Raja, 131–149. Cambridge, MA: The MIT Press. Scholar
  35. Cushman, Fiery. 2013. The functional design of punishment and the psychology of learning. In Psychological and environmental foundations of cooperation, Signaling, commitment and emotion, ed. Richard Joyce, Kim Sterelny, B. Calcott, and B. Fraser, vol. 2. Cambridge, MA: MIT Press.Google Scholar
  36. Cushman, Fiery, and Liane Young. 2011. Patterns of moral judgment derive from nonmoral psychological representations. Cognitive Science 35(6): 1052–1075. Scholar
  37. Cusimano, Corey, Stuti Thapa Magar, and Bertram F. Malle. 2017. Judgment before emotion: People access moral evaluations faster than affective states. In Proceedings of the 39th annual conference of the cognitive science society, ed. G. Gunzelmann, A. Howes, T. Tenbrink, and E. J. Davelaar, 1848–1853. Austin: Cognitive Science Society.Google Scholar
  38. Damasio, Antonio R. 1994. Descartes’ error: Emotion, reason, and the human brain. New York: Putnam.Google Scholar
  39. Darling, Kate, Palash Nandy, and Cynthia Breazeal. 2015. Empathic concern and the effect of stories in human-robot interaction. In Proceedings of the 24th IEEE international symposium on robot and human interactive communication (RO-MAN), 770–775. Kobe: IEEE.
  40. Davies, Mark. 2010. The corpus of contemporary American english as the first reliable monitor corpus of english. Literary and Linguistic Computing 25(4): 447–464. Scholar
  41. Davis, Tyler, Bradley C. Love, and W. Todd Maddox. 2009. Anticipatory emotions in decision tasks: Covert markers of value or attentional processes? Cognition 112(1): 195–200. Scholar
  42. Dennis, Louise, Michael Fisher, Marija Slavkovik, and Matt Webster. 2016. Formal verification of ethical choices in autonomous systems. Robotics and Autonomous Systems 77(March): 1–14. Scholar
  43. Dersley, Ian, and Anthony Wootton. 2000. Complaint sequences within antagonistic argument. Research on Language and Social Interaction 33(4): 375–406. Scholar
  44. Drew, Paul. 1998. Complaints about transgressions and misconduct. Research on Language & Social Interaction 31(3/4): 295–325.CrossRefGoogle Scholar
  45. Edwards, Carolyn Pope. 1987. Culture and the construction of moral values: A comparative ethnography of moral encounters in two cultural settings. In The emergence of morality in young children, ed. Jerome Kagan and Sharon Lamb, 123–151. Chicago: University of Chicago Press.Google Scholar
  46. Eisenberg, Nancy. 2000. Emotion, regulation, and moral development. Annual Review of Psychology 51: 665–697.CrossRefGoogle Scholar
  47. Fehr, Ernst, and Urs Fischbacher. 2004. Third-Party punishment and social norms. Evolution and Human Behavior 25(2): 63–87. Scholar
  48. Feinberg, Matthew, Joey T. Cheng, and Robb Willer. 2012. Gossip as an effective and low-cost form of punishment. Behavioral and Brain Sciences 35(01): 25–25. Scholar
  49. FeldmanHall, Oriel, Tim Dalgleish, Davy Evans, and Dean Mobbs. 2015. Empathic concern drives costly altruism. NeuroImage 105(January): 347–356. Scholar
  50. Flack, J. C., and Frans B. M. de Waal. 2000. Any animal whatever’. Darwinian building blocks of morality in monkeys and apes. Journal of Consciousness Studies 7(1–2): 1–29.Google Scholar
  51. Flanagan, Mary, Daniel C. Howe, and Helen Nissenbaum. 2008. Embodying values in technology: Theory and practice. In Information technology and moral philosophy, Cambridge studies in philosophy and public policy, ed. Jeroen van den Hoven and John Weckert, 322–353. Cambridge, UK: Cambridge University Press.
  52. Floridi, Luciano, and J. W. Sanders. 2004. On the morality of artificial agents. Minds and Machines 14(3): 349–379. Scholar
  53. Ford, Kenneth M., and Patrick J. Hayes. 1991. Reasoning agents in a dynamic world: The frame problem. Greenwich: JAI Press.Google Scholar
  54. Fridin, Marina. 2014. Kindergarten social assistive robot: First meeting and ethical issues. Computers in Human Behavior 30(January): 262–272. Scholar
  55. Funk, Michael, Bernhard Irrgang, and Silvio Leuteritz. 2016. Enhanced information warfare and three moral claims of combat drone responsibility. In Drones and responsibility: Legal, philosophical and socio-technical perspectives on remotely controlled weapons, ed. Ezio Di Nucci and Filippo Santoni de Sio, 182–196. London: Routledge.CrossRefGoogle Scholar
  56. Gawronski, Bertram, Paul Conway, Joel Armstrong, Rebecca Friesdorf, and Mandy Hütter. 2018. Effects of incidental emotions on moral dilemma judgments: An analysis using the CNI model. Emotion (February). Scholar
  57. Gentner, Dedre, Asli Ozyürek, Ozge Gürcanli, and Susan Goldin-Meadow. 2013. Spatial language facilitates spatial cognition: Evidence from children who lack language input. Cognition 127(3): 318–330. Scholar
  58. Gilbert, Elizabeth A., Elizabeth R. Tenney, Christopher R. Holland, and Barbara A. Spellman. 2015. Counterfactuals, control, and causation: Why knowledgeable people get blamed more. Personality and Social Psychology Bulletin, March, 0146167215572137. Scholar
  59. Governatori, Guido, and Antonino Rotolo. 2007. BIO logical agents: Norms, beliefs, intentions in defeasible logic. In Normative Multi-Agent Systems, ed. Guido Boella, Leon van der Torre, and Harko Verhagen. Dagstuhl seminar proceedings. Dagstuhl, Germany: Internationales Begegnungs- und Forschungszentrum für Informatik (IBFI), Schloss Dagstuhl, Germany.
  60. de Graaf, Maartje, and Bertram F. Malle. 2017. How people explain action (and autonomous intelligent systems should too). In 2017 AAAI Fall symposium series technical reports, FS-17-01, 19–26. Palo Alto: AAAI Press.Google Scholar
  61. Greene, Joshua D., R. B. Sommerville, Leigh E. Nystrom, John M. Darley, and Jonathan D. Cohen. 2001. An FMRI investigation of emotional engagement in moral judgment. Science 293(5537): 2105–2108. Scholar
  62. Greene, Joshua D., Leigh E. Nystrom, Andrew D. Engell, John M. Darley, and Jonathan D. Cohen. 2004. The neural bases of cognitive conflict and control in moral judgment. Neuron 44(2): 389–400. Scholar
  63. Guglielmo, Steve, and Bertram F. Malle. 2017. Information-acquisition processes in moral judgments of blame. Personality and Social Psychology Bulletin 43(7): 957–971. Scholar
  64. Guglielmo, Steve, Andrew E. Monroe, and Bertram F. Malle. 2009. At the heart of morality lies folk psychology. Inquiry: An Interdisciplinary Journal of Philosophy 52(5): 449–466. Scholar
  65. Gunkel, David J. 2014. A vindication of the rights of machines. Philosophy & Technology 27(1): 113–132. Scholar
  66. Haidt, Jonathan. 2001. The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review 108(4): 814–834. Scholar
  67. Hamlin, J. Kiley. 2013. Moral judgment and action in preverbal infants and toddlers: Evidence for an innate moral core. Current Directions in Psychological Science 22(3): 186–193. Scholar
  68. Harenski, Carla L., Keith A. Harenski, Matthew S. Shane, and Kent A. Kiehl. 2010. Aberrant neural processing of moral violations in criminal psychopaths. Journal of Abnormal Psychology 119(4): 863–874.CrossRefGoogle Scholar
  69. Harvey, Michael D., and Michael E. Enzle. 1981. A cognitive model of social norms for understanding the transgression–helping effect. Journal of Personality and Social Psychology 41(5): 866–875. Scholar
  70. Henrich, Joseph. 2009. The evolution of costly displays, cooperation and religion: Credibility enhancing displays and their implications for cultural evolution. Evolution and Human Behavior 30(4): 244–260. Scholar
  71. Herrmann, Patricia A., Cristine H. Legare, Paul L. Harris, and Harvey Whitehouse. 2013. Stick to the script: The effect of witnessing multiple actors on children’s imitation. Cognition 129(3): 536–543. Scholar
  72. Hilton, Denis J. 2007. Causal explanation: From social perception to knowledge-based causal attribution. In Social psychology: Handbook of basic principles, 2nd ed., ed. Arie W. Kruglanski and E. Tory Higgins, 232–253. New York: Guilford Press.Google Scholar
  73. Hofmann, Bjørn. 2013. Ethical challenges with welfare technology: A review of the literature. Science and Engineering Ethics 19(2): 389–406.CrossRefGoogle Scholar
  74. Horne, Zachary, and Derek Powell. 2016. How large is the role of emotion in judgments of moral dilemmas? PLOS ONE 11(7): e0154780. Scholar
  75. Hutcherson, Cendri A., and James J. Gross. 2011. The moral emotions: A social–functionalist account of anger, disgust, and contempt. Journal of Personality and Social Psychology 100(4): 719–737. Scholar
  76. Iba, W. F., and P. Langley. 2011. Exploring moral reasoning in a cognitive architecture. In Proceedings of the 33rd annual meeting of the cognitive science society. Austin, TX: Cognitive Science SocietyGoogle Scholar
  77. Joyce, Richard. 2006. The evolution of morality. Cambridge, MA: MIT Press.Google Scholar
  78. Kahn, Jr., Peter H., Takayuki Kanda, Hiroshi Ishiguro, Brian T. Gill, Jolina H. Ruckert, Solace Shen, Heather E. Gary, Aimee L. Reichert, Nathan G. Freier, and Rachel L. Severson. 2012. Do people hold a humanoid robot morally accountable for the harm it causes? In Proceedings of the seventh annual ACM/IEEE international conference on human-robot interaction, 33–40. New York: ACM.
  79. Kasenberg, Daniel, and Matthias Scheutz. 2018. Norm conflict resolution in stochastic domains. In Proceedings of the thirty-second AAAI conference on artificial intelligence, 32–36. Palo Alto, CA: AAAI Press.Google Scholar
  80. Kenett, Yoed N., M. M. Allaham, Joseph L. Austerweil, and Bertram F. Malle. 2016. The norm fluency task: Unveiling the properties of norm representation. In Poster presented at the 57th annual meeting of the psychonomic society, Boston, MA, November 2016. Boston.Google Scholar
  81. Knobe, Joshua, and B. Fraser. 2008. Causal judgment and moral judgment: Two experiments. In Moral psychology (Vol. 2): The cognitive science of morality: Intuition and diversity, 2, 441–447 Cambridge, MA: MIT Press.Google Scholar
  82. Kohlberg, Lawrence. 1984. The psychology of moral development: The nature and validity of moral stages. San Francisco: Harper & Row.Google Scholar
  83. Koop, Gregory J. 2013. An assessment of the temporal dynamics of moral decisions. Judgment and Decision making 8(5): 527–539.Google Scholar
  84. Levy, David. 2015. When robots do wrong. In Cognitive robotics, ed. Hooman Samani, 3–22. Boca Raton: CRC Press.
  85. Malle, Bertram F. 1999. How people explain behavior: A new theoretical framework. Personality and Social Psychology Review 3(1): 23–48. Scholar
  86. Malle, Bertram F. 2011. Time to give up the dogmas of attribution: A new theory of behavior explanation. In Advances of experimental social psychology, ed. Mark P. Zanna and James M. Olson, vol. 44, 297–352. San Diego: Academic Press.Google Scholar
  87. Malle, Bertram F. 2016. Integrating robot ethics and machine morality: The study and design of moral competence in robots. Ethics and Information Technology 18(4): 243–256. Scholar
  88. Malle, Bertram F. 2018. From binary deontics to deontic continua: The nature of human (and Robot) norm systems. Paper presented at the third international robo-philosophy conference, University of Vienna, Austria.Google Scholar
  89. Malle, Bertram F., and Jess Holbrook. 2012. Is there a hierarchy of social inferences? The likelihood and speed of inferring intentionality, mind, and personality. Journal of Personality and Social Psychology 102(4): 661–684. Scholar
  90. Malle, Bertram F., and Matthias Scheutz. 2014. Moral competence in social robots. In Proceedings of IEEE international symposium on ethics in engineering, science, and technology, Ethics’2014, 30–35. Chicago: IEEE.Google Scholar
  91. Malle, Bertram F., Steve Guglielmo, and Andrew E. Monroe. 2014. A theory of blame. Psychological Inquiry 25(2): 147–186. Scholar
  92. Malle, Bertram F., Matthias Scheutz, Thomas Arnold, John Voiklis, and Corey Cusimano. 2015. Sacrifice one for the good of many? People apply different moral norms to human and robot agents. In Proceedings of the tenth annual ACM/IEEE international conference on Human-Robot Interaction, HRI’15, 117–124. New York: ACM.Google Scholar
  93. Malle, Bertram F., Matthias Scheutz, and Joseph L. Austerweil. 2017. Networks of social and moral norms in human and robot agents. In A world with robots: International Conference on Robot Ethics: ICRE 2015, ed. Maria Isabel Aldinhas Ferreira, Joao Silva Sequeira, Mohammad Osman Tokhi, Endre E. Kadar, and Gurvinder Singh Virk, 3–17. Cham: Springer International Publishing. Scholar
  94. Matthias, Andreas. 2004. The responsibility gap: Ascribing responsibility for the actions of learning automata. Ethics and Information Technology 6(3): 175–183. Scholar
  95. May, Joshua. 2018. Précis of regard for reason in the moral mind. Behavioral and Brain Sciences.Google Scholar
  96. McKenna, Michael. 2012. Directed blame and conversation. In Blame: Its nature and norms, ed. D. Justin Coates and Neal A. Tognazzini, 119–140. New York: Oxford University Press.CrossRefGoogle Scholar
  97. McNeill, William Hardy. 1995. Keeping together in time: Dance and drill in human history. Cambridge, MA: Harvard University Press Scholar
  98. MHAT-IV. 2006. Mental Health Advisory Team (MHAT) IV: Operation Iraqi freedom 05-07 final report. Washington, DC: Office of the Surgeon, Multinational Force-Iraq; Office of the Surgeon General, United States Army Medical Command.Google Scholar
  99. Mikhail, John. 2007. Universal moral grammar: Theory, evidence and the future. Trends in Cognitive Sciences 11(4): 143–152. Scholar
  100. Milgram, Stanley, Leonard Bickman, and Lawrence Berkowitz. 1969. Note on the drawing power of crowds of different size. Journal of Personality and Social Psychology 13(2): 79–82. Scholar
  101. Miller, Keith W., Marty J. Wolf, and Frances Grodzinsky. 2017. This ‘Ethical Trap’ is for roboticists, not robots: On the issue of artificial agent ethical decision-making. Science and Engineering Ethics 23(2): 389–401. Scholar
  102. Monin, Benoît, David A. Pizarro, and Jennifer S. Beer. 2007. Deciding versus reacting: Conceptions of moral judgment and the reason-affect debate. Review of General Psychology 11(2): 99–111. Scholar
  103. Monroe, Andrew E., and Bertram F. Malle. 2018. People systematically update moral judgments of blame. Journal of Personality and Social Psychology.
  104. Monroe, Andrew E., Kyle D. Dillon, and Bertram F. Malle. 2014. Bringing free will down to earth: People’s psychological concept of free will and its role in moral judgment. Consciousness and Cognition 27(July): 100–108. Scholar
  105. Moor, James H. 2006. The nature, importance, and difficulty of machine ethics. IEEE Intelligent Systems 21(4): 18–21. Scholar
  106. Moretto, Giovanna, Elisabetta Làdavas, Flavia Mattioli, and Giuseppe di Pellegrino. 2010. A psychophysiological investigation of moral judgment after ventromedial prefrontal damage. Journal of Cognitive Neuroscience 22(8): 1888–1899. Scholar
  107. Mullan, John. 2017. We need robots to have morals. Could shakespeare and austen help? The Guardian, July 24, 2017, sec. Opinion.
  108. Nichols, Shaun, and Ron Mallon. 2006. Moral dilemmas and moral rules. Cognition 100(3): 530–542. Scholar
  109. Nisbett, R. E., and T. D. Wilson. 1977. Telling more than we know: Verbal reports on mental processes. Psychological Review 84: 231–259.CrossRefGoogle Scholar
  110. Parthemore, Joel, and Blay Whitby. 2013. What makes any agent a moral agent? Reflections on machine consciousness and moral agency. International Journal of Machine Consciousness 4: 105–129.CrossRefGoogle Scholar
  111. Paxton, Joseph M., Leo Ungar, and Joshua D. Greene. 2012. Reflection and reasoning in moral judgment. Cognitive Science 36(1): 163–177. Scholar
  112. Pereira, Luís Moniz, and Ari Saptawijaya. 2007. Modelling morality with prospective logic. In Progress in artificial intelligence, Lecture Notes in Computer Science, ed. José Neves, Manuel Filipe Santos, and José Manuel Machado, 99–111. Springer Berlin/Heidelberg.
  113. Petersen, Stephen. 2007. The ethics of robot servitude. Journal of Experimental & Theoretical Artificial Intelligence 19(1): 43–54. Scholar
  114. Pizarro, David A., Yoel Inbar, and Chelsea Helion. 2011. On disgust and moral judgment. Emotion Review 3(3): 267–268. Scholar
  115. Purves, Duncan, Ryan Jenkins, and Bradley J. Strawser. 2015. Autonomous machines, moral judgment, and acting for the right reasons. Ethical Theory and Moral Practice 18(4): 851–872. Scholar
  116. Rand, David G., and Martin A. Nowak. 2013. Human cooperation. Trends in Cognitive Sciences 17(8): 413–425. Scholar
  117. Riedl, Mark O., and Brent Harrison. 2016. Using stories to teach human values to artificial agents. In AI, ethics, and society, papers from the 2016 AAAI Workshop, Phoenix, Arizona, USA, February 13, 2016.
  118. Rossano, Matt J. 2012. The essential role of ritual in the transmission and reinforcement of social norms. Psychological Bulletin 138(3): 529–549. Scholar
  119. Royzman, Edward B., Geoffrey P. Goodwin, and Robert F. Leeman. 2011. When sentimental rules collide: ‘Norms with feelings’ in the dilemmatic context. Cognition 121(1): 101–114. Scholar
  120. Sarathy, Vasanth, Matthias Scheutz, Yoed N. Kenett, M. M. Allaham, Joseph L. Austerweil, and Bertram F. Malle. 2017a. Mental representations and computational modeling of context-specific human norm systems. In Proceedings of the 39th annual meeting of the Cognitive Science Society, London, 1035–1040. Austin: Cognitive Science Society.Google Scholar
  121. Sarathy, Vasanth, Matthias Scheutz, and Bertram F. Malle. 2017b. Learning behavioral norms in uncertain and changing contexts. In Proceedings of the 2017 8th IEEE international conference on cognitive infocommunications (CogInfoCom), 301–306. Pascataway, NJ: IEEE Press.Google Scholar
  122. Sarathy, Vasanth, Bradley Oosterveld, Evan Krause, and Matthias Scheutz. 2018. Learning cognitive affordances from natural language instructions. Advances in Cognitive Systems 6: 1–20.Google Scholar
  123. Saucier, Gerard, and Lewis R. Goldberg. 1996. The language of personality: Lexical perspectives on the five-factor model. In The five-factor model of personality: Theoretical perspectives, ed. Jerry S. Wiggins, 21–50. New York: Guilford Press.Google Scholar
  124. Scheutz, Matthias. 2012. The affect dilemma for artificial agents: Should we develop affective artificial agents? IEEE Transactions on Affective Computing 3(4): 424–433.CrossRefGoogle Scholar
  125. Scheutz, Matthias. 2014. The need for moral competency in autonomous agent architectures In Fundamental issues of artificial intelligence, ed. Vincent C. Müller, 515–525 Berlin: Springer.Google Scholar
  126. Scheutz, Matthias, and Bertram F. Malle. 2014. ‘Think and do the right thing’: A plea for morally competent autonomous robots. In Proceedings of the IEEE international symposium on ethics in engineering, science, and technology, Ethics’2014, 36–39. Red Hook: Curran Associates/IEEE Computer Society.Google Scholar
  127. Scheutz, Matthias, and Bertram F. Malle. 2017. Moral Robots. In The routledge handbook of neuroethics, ed. L. Syd M. Johnson and Karen Rommelfanger, 363–377. New York: Routledge.CrossRefGoogle Scholar
  128. Scheutz, Matthias, Bertram F. Malle, and Gordon Briggs. 2015. Towards morally sensitive action selection for autonomous social robots In Proceedings of the 24th IEEE international symposium on robot and human interactive communication (RO-MAN), 492–497. Kobe: IEEE Press.Google Scholar
  129. Schmidt, Marco F. H., Hannes Rakoczy, and Michael Tomasello. 2011. Young children attribute normativity to novel actions without pedagogy or normative language. Developmental Science 14(3): 530–539.CrossRefGoogle Scholar
  130. Searle, John R. 1980. Minds, brains, and programs. Behavioral and Brain Sciences 3(3): 417–457.CrossRefGoogle Scholar
  131. Semin, Gün R., and A. S. R Manstead. 1983. The accountability of conduct: A social psychological analysis. London: Academic Press.Google Scholar
  132. Shafer, Glenn. 1976. A mathematical theory of evidence. Princeton: Princeton University Press.Google Scholar
  133. Sparrow, Robert. 2007. Killer robots. Journal of Applied Philosophy 24(1): 62–77. Scholar
  134. Sripada, Chandra Sekhar, and Stephen Stich. 2006. A framework for the psychology of norms. In The innate mind (Vol. 2: Culture and cognition), ed. Peter Carruthers, Stephen Laurence, and Stephen Stich, 280–301. New York: Oxford University Press.CrossRefGoogle Scholar
  135. Stahl, Bernd Carsten. 2004. Information, ethics, and computers: The problem of autonomous moral agents. Minds and Machines 14(1): 67–83. Scholar
  136. Sullins, John P. 2011. Introduction: Open questions in roboethics. Philosophy & Technology 24(3): 233. Scholar
  137. Todd, Ben. 2011. ‘Non, Non! you can’t wee wee here, monsieur’: Gerard depardieu gets caught short in a plane. DailyMail.Com. August 18, 2011.
  138. Tomasello, Michael, and Amrisha Vaish. 2013. Origins of human cooperation and morality. Annual Review of Psychology 64(1): 231–255. Scholar
  139. Ullmann-Margalit, Edna. 1977. The emergence of norms, Clarendon Library of Logic and Philosophy. Oxford: Clarendon Press.Google Scholar
  140. de Villiers, Jill. 2007. The interface of language and theory of mind. Lingua. International Review of General Linguistics. Revue Internationale De Linguistique Generale 117(11): 1858–1878. Scholar
  141. Voiklis, John, Corey Cusimano, and Bertram F. Malle. 2014. A social-conceptual map of moral criticism. In Proceedings of the 36th annual conference of the cognitive science society, ed. Paul Bello, M. Guarini, M. McShane, and Brian Scassellati, 1700–1705. Austin: Cognitive Science Society.Google Scholar
  142. Voiklis, John, Corey Cusimano, and Bertram F. Malle. 2016. Using moral communication to reveal moral cognition. Paper presented at the international conference on thinking, providence, RI.Google Scholar
  143. Walker, Margaret Urban. 2006. Moral repair: Reconstructing moral relations after wrongdoing. New York: Cambridge University Press.CrossRefGoogle Scholar
  144. Wallach, Wendell, and Colin Allen. 2008. Moral machines: Teaching robots right from wrong. New York: Oxford University Press.Google Scholar
  145. Wang, Zhouxia, Tianshui Chen, Jimmy Ren, Weihao Yu, Hui Cheng, and Liang Lin. 2018. Deep reasoning with knowledge graph for social relationship understanding. ArXiv:1807.00504 [Cs], July.
  146. Weingart, Laurie R., Kristin J. Behfar, Corinne Bendersky, Gergana Todorova, and Karen A. Jehn. 2014. The directness and oppositional intensity of conflict expression. Academy of Management Review 40(2): 235–262. Scholar
  147. Wenk, Gary Lee. 2015. Your brain on food: How chemicals control your thoughts and feelings. New York: Oxford University Press.Google Scholar
  148. West, John. 2016. Microsoft’s disastrous tay experiment shows the hidden dangers of AI. Quartz, April 2, 2016.
  149. Westmarland, Louise. 2005. Police ethics and integrity: Breaking the blue code of silence. Policing & Society 15(2): 145–165.CrossRefGoogle Scholar
  150. Wright, Jennifer Cole, and Karen Bartsch. 2008. Portraits of early moral sensibility in two children’s everyday conversations. Merrill-Palmer Quarterly 54(1): 56–85.
  151. van Wynsberghe, Aimee. 2013. Designing robots for care: Care centered value-sensitive design. Science and Engineering Ethics 19(2): 407–433. Scholar

Copyright information

© Springer Fachmedien Wiesbaden GmbH, ein Teil von Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Cognitive, Linguistic, and Psychological SciencesBrown UniversityProvidenceUSA
  2. 2.Department of Computer ScienceTufts UniversityMedfordUSA

Personalised recommendations