Sensitivity to Risk Profiles of Users When Developing AI Systems

  • Robin Cohen
  • Rishav Raj AgarwalEmail author
  • Dhruv Kumar
  • Alexandre Parmentier
  • Tsz Him Leung
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12109)


The AI community today has renewed concern about the social implications of the models they design, imagining the impact of deployed systems. One thrust has been to reflect on issues of fairness and explainability before the design process begins. There is increasing awareness as well of the need to engender trust from users, examining the origins of mistrust as well as the value of multiagent trust modelling solutions. In this paper, we argue that social AI efforts to date often imagine a homogenous user base and those models which do support differing solutions for users with different profiles have not yet examined one important consideration upon which trusted AI may depend: the risk profile of the user. We suggest how user risk attitudes can be integrated into approaches that try to reason about such dilemmas as sacrificing optimality for the sake of explainability. In the end, we reveal that it is challenging to be satisfying the myriad needs of users in their desire to be more comfortable accepting AI solutions and conclude that tradeoffs need to be examined and balanced. We advocate reasoning about these tradeoffs concerning user models and risk profiles, as we design the decision making algorithms of our systems.


Position paper Trusted AI Risk profiles Explainability 


  1. 1.
    Anjomshoae, S., Främling, K., Najjar, A.: Explanations of black-box model predictions by contextual importance and utility. In: Calvaresi, D., Najjar, A., Schumacher, M., Främling, K. (eds.) EXTRAAMAS 2019. LNCS (LNAI), vol. 11763, pp. 95–109. Springer, Cham (2019). Scholar
  2. 2.
    Chakraborti, T., Kulkarni, A., Sreedharan, S., Smith, D.E., Kambhampati, S.: Explicability? legibility? predictability? transparency? privacy? security? the emerging landscape of interpretable agent behavior. In: Proceedings of the International Conference on Automated Planning and Scheduling, vol. 29, pp. 86–96 (2019)Google Scholar
  3. 3.
    Charness, G., Gneezy, U., Imas, A.: Experimental methods: eliciting risk preferences. J. Econ. Behav. Organ. 87, 43–51 (2013)CrossRefGoogle Scholar
  4. 4.
    Cohen, R., Schaekermann, M., Liu, S., Cormier, M.: Trusted AI and the contribution of trust modeling in multiagent systems. In: Proceedings of AAMAS, pp. 1644–1648 (2019)Google Scholar
  5. 5.
    Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., Huq, A.: Algorithmic decision making and the cost of fairness. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 797–806. ACM (2017)Google Scholar
  6. 6.
    Dodge, J., Liao, Q.V., Zhang, Y., Bellamy, R.K., Dugan, C.: Explaining models: an empirical study of how explanations impact fairness judgment. In: Proceedings of the 24th International Conference on Intelligent User Interfaces, pp. 275–285. ACM (2019)Google Scholar
  7. 7.
    Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 214–226. ACM (2012)Google Scholar
  8. 8.
    Hardt, M., Price, E., Srebro, N., et al.: Equality of opportunity in supervised learning. In: Advances in Neural Information Processing Systems, pp. 3315–3323 (2016)Google Scholar
  9. 9.
    Hashimoto, T.B., Srivastava, M., Namkoong, H., Liang, P.: Fairness without demographics in repeated loss minimization. arXiv preprint arXiv:1806.08010 (2018)
  10. 10.
    Hellström, T., Bensch, S.: Understandable robots-what, why, and how. Paladyn J. Behav. Robot. 9(1), 110–123 (2018)CrossRefGoogle Scholar
  11. 11.
    Hines, G., Larson, K.: Preference elicitation for risky prospects. In: Proceedings of AAMAS, pp. 889–896 (2010)Google Scholar
  12. 12.
    Kambhampati, S.: Synthesizing explainable behavior for human-ai collaboration. In: Proceedings of AAMAS. Richland, SC, pp. 1–2 (2019)Google Scholar
  13. 13.
    Kass, R., Finin, T.: Modeling the user in natural language systems. Comput. Linguist. 14(3), 5–22 (1988)Google Scholar
  14. 14.
    Mayer, R.C., Davis, J.H., Schoorman, F.D.: An integrative model of organizational trust. Acad. Manag. Rev. 20(3), 709–734 (1995)CrossRefGoogle Scholar
  15. 15.
    Melo, C.D., Marsella, S., Gratch, J.: People do not feel guilty about exploiting machines. ACM Trans. Comput. Hum. Interac. (TOCHI) 23(2), 8 (2016)CrossRefGoogle Scholar
  16. 16.
    de Melo, C.M., Marsella, S., Gratch, J.: Do as I say, not as I do: challenges in delegating decisions to automated agents. In: Proceedings of AAMAS, pp. 949–956 (2016)Google Scholar
  17. 17.
    Nomura, T., Kawakami, K.: Relationships between robot’s self-disclosures and human’s anxiety toward robots. In: Proceedings of the 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology-vol. 03, pp. 66–69. IEEE Computer Society (2011)Google Scholar
  18. 18.
    Rossi, A., Dautenhahn, K., Koay, K.L., Walters, M.L.: The impact of peoples’ personal dispositions and personalities on their trust of robots in an emergency scenario. Paladyn J. Behav. Robot. 9(1), 137–154 (2018)CrossRefGoogle Scholar
  19. 19.
    Rossi, A., Holthaus, P., Dautenhahn, K., Koay, K.L., Walters, M.L.: Getting to know pepper: effects of people’s awareness of a robot’s capabilities on their trust in the robot. In: Proceedings of the 6th International Conference on Human-Agent Interaction, pp. 246–252. ACM (2018)Google Scholar
  20. 20.
    Salem, M., Lakatos, G., Amirabdollahian, F., Dautenhahn, K.: Would you trust a (faulty) robot?: effects of error, task type and personality on human-robot cooperation and trust. In: Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, pp. 141–148. ACM (2015)Google Scholar
  21. 21.
    Sengupta, S., Zahedi, Z., Kambhampati, S.: To monitor or to trust: observing robot’s behavior based on a game-theoretic model of trust. In: Proceedings of the Trust Workshop at AAMAS (2019)Google Scholar
  22. 22.
    Sreedharan, S., Kambhampati, S., et al.: Balancing explicability and explanation in human-aware planning. In: 2017 AAAI Fall Symposium Series (2017)Google Scholar
  23. 23.
    Tran, T.T., Cohen, R., Langlois, E., Kates, P.: Establishing trust in multiagent environments: realizing the comprehensive trust management dream. TRUST@ AAMAS 1740, 35–43 (2014)Google Scholar
  24. 24.
    Tversky, A., Kahneman, D.: Advances in prospect theory: cumulative representation of uncertainty. J. Risk Uncertainty 5(4), 297–323 (1992)CrossRefGoogle Scholar
  25. 25.
    Yuksel, B.F., Collisson, P., Czerwinski, M.: Brains or beauty: how to engender trust in user-agent interactions. ACM Trans. Internet Technol. (TOIT) 17(1), 2 (2017)CrossRefGoogle Scholar
  26. 26.
    Zahedi, Z., Olmo, A., Chakraborti, T., Sreedharan, S., Kambhampati, S.: Towards understanding user preferences for explanation types in model reconciliation. In: 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 648–649. IEEE (2019)Google Scholar
  27. 27.
    Zhao, J., et al.: Men also like shopping: reducing gender bias amplification using corpus-level constraints. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Robin Cohen
    • 1
  • Rishav Raj Agarwal
    • 1
    Email author
  • Dhruv Kumar
    • 1
  • Alexandre Parmentier
    • 1
  • Tsz Him Leung
    • 1
  1. 1.School of Computer ScienceUniversity of WaterlooWaterlooCanada

Personalised recommendations