Advertisement

Philosophy & Technology

, Volume 32, Issue 4, pp 661–683 | Cite as

Transparency in Algorithmic and Human Decision-Making: Is There a Double Standard?

  • John ZerilliEmail author
  • Alistair Knott
  • James Maclaurin
  • Colin Gavaghan
Research Article

Abstract

We are sceptical of concerns over the opacity of algorithmic decision tools. While transparency and explainability are certainly important desiderata in algorithmic governance, we worry that automated decision-making is being held to an unrealistically high standard, possibly owing to an unrealistically high estimate of the degree of transparency attainable from human decision-makers. In this paper, we review evidence demonstrating that much human decision-making is fraught with transparency problems, show in what respects AI fares little worse or better and argue that at least some regulatory proposals for explainable AI could end up setting the bar higher than is necessary or indeed helpful. The demands of practical reason require the justification of action to be pitched at the level of practical reason. Decision tools that support or supplant practical reasoning should not be expected to aim higher than this. We cast this desideratum in terms of Daniel Dennett’s theory of the “intentional stance” and argue that since the justification of action for human purposes takes the form of intentional stance explanation, the justification of algorithmic decisions should take the same form. In practice, this means that the sorts of explanations for algorithmic decisions that are analogous to intentional stance explanations should be preferred over ones that aim at the architectural innards of a decision tool.

Keywords

Algorithmic decision-making Transparency Explainable AI Intentional stance 

Notes

Acknowledgments

The authors wish to thank the participants of two roundtables, one held in Oxford, November 23–24, 2017, in partnership with the Uehiro Centre for Practical Ethics, University of Oxford, and the other in Dunedin, December 11–12, at the University of Otago.

Funding

This research was supported by a New Zealand Law Foundation grant (2016/ILP/10).

Compliance with Ethical Standards

Conflict of Interest

AK works for Soul Machines Ltd under contract. JZ, JM, and CG have no other disclosures or relevant affiliations apart from the appointments above.

References

  1. Allport, G. W. (1954). The nature of prejudice. Cambridge: Addison-Wesley.Google Scholar
  2. Angie, A. D., Connelly, S., Waples, E. P., & Kligyte, V. (2011). The influence of discrete emotions on judgement and decision-making: a meta-analytic review. Cognition and Emotion, 25(8), 1393–1422.CrossRefGoogle Scholar
  3. Aronson, & Dyer. (2013). Judicial review of administrative action (5th ed.). Sydney: Lawbook Co..Google Scholar
  4. Baker, J. H. (2002). An introduction to English legal history (4th ed.). New York: Oxford University Press.Google Scholar
  5. Barocas, S., & Selbst, A. D. (2015). Big data’s disparate impact. California Law Review, 104, 671–732.Google Scholar
  6. Begby, E. (2013). The epistemology of prejudice. Thought, 2(2), 90–99.Google Scholar
  7. Bezrukova, K., Spell, C. S., Perry, J. L., & Jehn, K. A. (2016) A meta-analytical integration of over 40 years of research on diversity training evaluation. Available at: http://scholarship.sha.cornell.edu/articles/974.
  8. Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J. & Shadbolt, N. (2018) “It’s reducing a human being to a percentage”: perceptions of justice in algorithmic decisions. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. New York: ACM.Google Scholar
  9. Burrell, J. (2016). How the machine “thinks”: understanding opacity in machine learning algorithms. Big Data and Society, 3(1), 1–12.CrossRefGoogle Scholar
  10. Cane, P. (2011). Administrative law (5th ed.). New York: Oxford University Press.Google Scholar
  11. Chopra, S., & White, L. F. (2011). A legal theory for autonomous artificial agents. Ann Arbor: University of Michigan Press.CrossRefGoogle Scholar
  12. Churchland, P. A. (1981). Eliminative materialism and the propositional attitudes. Journal of Philosophy, 78, 67–90.Google Scholar
  13. Coglianese, C., & Lazer, D. (2003). Management-based regulation: prescribing private management to achieve public goals. Law and Society Review, 37(4), 691–730.CrossRefGoogle Scholar
  14. Corbett-Davies, S., Pierson, E., Feller, A., Goel, S. & Huq, A. (2016) Algorithmic decision making and the cost of fairness. Proceedings of KDD’17. Available at: https://arxiv.org/pdf/1701.08230.pdf.
  15. Corbett-Davies, S., Pierson, E., Feller, A. & Goel, S. (2017) A computer program used for bail and sentencing decisions was labeled biased against blacks. It’s actually not that clear. Washington Post.Google Scholar
  16. Crawford, K. (2016) Artificial intelligence’s white guy problem. New York Times.Google Scholar
  17. Crawford, K., & Calo, R. (2016). There is a blind spot in AI research. Nature, 538, 311–313.CrossRefGoogle Scholar
  18. Damasio, A. R. (1994). Descartes’ error: emotion, reason, and the human brain. New York: Putnam’s Sons.Google Scholar
  19. Danaher, J., Hogan, M. J., Noone, C., Kennedy, R., Behan, A., De Paor, A., Felzmann, H., Haklay, M., Khoo, S., Morison, J., Murphy, M. H., O’Brolchain, N., Schafer, B., & Shankar, K. (2017). Algorithmic governance: developing a research agenda through the power of collective intelligence. Big Data and Society, 1–21.Google Scholar
  20. Dennett, D. (1987). The intentional stance. Cambridge: MIT Press.Google Scholar
  21. Dennett, D. (1991). Real patterns. Journal of Philosophy, 87, 27–51.CrossRefGoogle Scholar
  22. Dennett, D. (1995). Darwin’s dangerous idea: evolution and the meanings of life. New York: Simon & Schuster.Google Scholar
  23. Diakopoulos, N. (2015). Algorithmic accountability: journalistic investigation of computational power structures. Digital Journalism, 3(3), 398–415.CrossRefGoogle Scholar
  24. Dutta, S. (2017) Do computers make better bank managers than humans? The Conversation.Google Scholar
  25. Dworkin, R. (1977). Taking rights seriously. London: Duckworth.Google Scholar
  26. Dworkin, R. (1986). Law’s empire. London: Fontana Books.Google Scholar
  27. Edwards, L., & Veale, M. (2017). Slave to the algorithm? Why a “right to an explanation” is probably not the remedy you are looking for. Duke Law and Technology Review, 16(1), 18–84.Google Scholar
  28. Edwards, L. & Veale, M. (2018) Enslaving the algorithm: From a “right to an explanation” to a “right to better decisions”? IEEE Security & Privacy.Google Scholar
  29. Erdélyi, O.J. & Goldsmith, J. (2018) Regulating artificial intelligence: proposal for a global solution. AAAI/ACM Conference on Artificial Intelligence, Ethics and Society. Available at: http://www.aiesconference.com/wpcontent/papers/main/AIES_2018_paper_13.pdf.
  30. Eubanks, V. (2017). Automating inequality: How high-tech tools profile, police, and punish the poor. New York: St Martin’s Press.Google Scholar
  31. Fodor, J. A. (1981). Three cheers for propositional attitudes. In J. A. Fodor (Ed.), RePresentations: philosophical essays on the foundations of cognitive science. Cambridge: MIT Press.Google Scholar
  32. Forssbæck, J., & Oxelheim, L. (2014). The multifaceted concept of transparency. In J. Forssbæck & L. Oxelheim (Eds.), The Oxford handbook of economic and institutional transparency (pp. 3–31). New York: Oxford University Press.CrossRefGoogle Scholar
  33. Friedman, B., & Nissenbaum, H. (1996). Bias in computer systems. ACM Transactions on Information Systems, 14(3), 330–347.CrossRefGoogle Scholar
  34. Grice, H. P. (1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and semantics 3: speech acts (pp. 41–58). New York: Academic Press.Google Scholar
  35. Griffiths, J. (2016) New Zealand passport robot thinks this Asian man’s eyes are closed. CNN.com December 9, 2016.
  36. Hardt, M., Price, E. & Srebro, N. (2016) Equality of opportunity in supervised learning. 30th Conference on Neural Information Processing Systems (NIPS 2016). Available at: https://arxiv.org/pdf/1610.02413v1.pdf.
  37. Heald, D. (2006). Transparency as an instrumental value. In C. Hood & D. Heald (Eds.), Transparency: the key to better governance? (pp. 59–73). Oxford: Oxford University Press.Google Scholar
  38. Hilton, D. J. (1990). Conversational processes and causal explanation. Psychological Bulletin, 107(1), 65–81.CrossRefGoogle Scholar
  39. Johnson, J.A. (2006). Technology and pragmatism: from value neutrality to value criticality. SSRN Scholarly Paper, Rochester, NY: Social Science Research Network. Available at: http://papers.ssrn.com/abstract=2154654.
  40. Kleinberg, J., Mullainathan, S. & Raghavan, M. (2017). Inherent trade-offs in the fair determination of risk scores. 8th Conference on Innovations in Theoretical Computer Science (ITCS 2017). Available at: https://arxiv.org/pdf/1609.05807.pdf.
  41. Klingele, C. (2016). The promises and perils of evidence-based corrections. Notre Dame Law Review, 91(2), 537–584.Google Scholar
  42. Langer, E., Blank, A. E., & Chanowitz, B. (1978). The mindlessness of ostensibly thoughtful action: the role of “placebic” information in interpersonal interaction. Journal of Personality and Social Psychology, 36(6), 635–642.CrossRefGoogle Scholar
  43. Larson, J., Mattu, S., Kirchner, L. & Angwin, J. (2016) How we analyzed the COMPAS recidivism algorithm. ProPublica.org May 23, 2016.
  44. Leslie, S. (2017). The original sin of cognition: race, prejudice and generalization. Journal of Philosophy, 114(8), 393–421.CrossRefGoogle Scholar
  45. Levendowski, A. (2017) How copyright law can fix artificial intelligence’s implicit bias problem. Washington Law Review (forthcoming). Available at: https://ssrn.com/abstract=3024938.
  46. Lombrozo, T. (2011). The instrumental value of explanations. Philosophy Compass, 6, 539.CrossRefGoogle Scholar
  47. Lum, K. & Isaac, W. (2016) To predict and serve? Bias in police-recorded data. Significance, 14–19.Google Scholar
  48. McEwen, R., Eldridge, J., & Caruso, D. (2018). Differential or deferential to media? The effect of prejudicial publicity on judge or jury. International Journal of Evidence and Proof, 22(2), 124–143.CrossRefGoogle Scholar
  49. Miller, T. (2017) Explanation in artificial intelligence: insights from the social sciences. Available at: https://arxiv.org/pdf/1706.07269.pdf.
  50. Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016). The ethics of algorithms: mapping the debate. Big Data and Society, 16, 1–21.Google Scholar
  51. Montavon, G., Bach, S., Binder, A., Samek, W., & Müller, K.-R. (2017). Explaining nonlinear classification decisions with Deep Taylor decomposition. Pattern Recognition, 65, 211.CrossRefGoogle Scholar
  52. Muehlhauser (2013) Transparency in safety-critical systems. Intelligence.org August 15, 2013. Available at: https://intelligence.org/2013/08/25/transparency-in-safety-critical-systems/.
  53. Nusser, S. (2009). Robust learning in safety-related domains: machine learning methods for solving safety-related application problems. Doctoral dissertation, Otto-von-Guericke-Universita ̈t Magdeburg. Available at: https://pdfs.semanticscholar.org/48c2/e5641101a4e5250ad903828c02025d269a1a.pdf.
  54. Oliver, W. M., & Batra, R. (2015). Standards of legitimacy in criminal negotiations. Harvard Negotiation Law Review, 20, 61–120.Google Scholar
  55. Oswald, M. & Grace, J. (2016). Intelligence, policing and the use of algorithmic analysis: A freedom of information-based study. Journal of Information Rights, Policy and Practice, 1(1). Available at: https://journals.winchesteruniversitypress.org/index.php/jirpp/article/view/16.
  56. Pasquale, F. (2014). The black box society: the secret algorithms that control money and information. Cambridge: Harvard University Press.Google Scholar
  57. Piattelli-Palmarini, M. (1995). La r’eforme du jugement ou comment ne plus se tromper. Paris: Odile Jacob.Google Scholar
  58. Plous, S. (2003a). The psychology of prejudice, stereotyping, and discrimination. In S. Plous (Ed.), Understanding prejudice and discrimination (pp. 3–48). New York: McGraw-Hill.Google Scholar
  59. Plous, S. (2003b). Understanding prejudice and discrimination. New York: McGraw-Hill.Google Scholar
  60. Pohl, J. (2008). Cognitive elements of human decision making Jens. In G. Phillips-Wren, N. Ichalkaranje, & L. C. Jain (Eds.), Intelligent decision making: an AI-based approach (pp. 3–40). Berlin: Springer.Google Scholar
  61. Pomerol, J.-C., & Adam, F. (2008). Understanding human decision making: a fundamental step towards effective intelligent decision support. In G. Phillips-Wren, N. Ichalkaranje, & L. C. Jain (Eds.), Intelligent decision making: an AI-based approach (pp. 41–76). Berlin: Springer.Google Scholar
  62. Prat, A. (2006). The more closely we are watched, the better we behave? In C. Hood & D. Heald (Eds.), Transparency: the key to better governance? (pp. 91–103). Oxford: Oxford University Press.Google Scholar
  63. Rosch, E. (1978). Principles of categorization. In E. Rosch & B. B. Lloyd (Eds.), Cognition and categorization (pp. 27–48). Hillsdale: Lawrence Erlbaum Associates.Google Scholar
  64. Schwab, K. (2016). The fourth industrial revolution. Geneva: Crown.Google Scholar
  65. Stephan, W. G., & Finlay, K. (1999). The role of empathy in improving intergroup relations. Journal of Social Issues, 55(4), 729–743.CrossRefGoogle Scholar
  66. Stich, S. (1983). From folk psychology to cognitive science. Cambridge: MIT Press.Google Scholar
  67. Tatman, R. (2016) Google’s speech recognition has a gender bias. Making Noise and Hearing Things.Google Scholar
  68. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: heuristics and biases. Science, 185, 1124–1131.CrossRefGoogle Scholar
  69. Van Otterlo, M. (2013). A machine learning view on profiling. In M. Hildebrandt & K. de Vries (Eds.), Privacy, due process and the computational turn: philosophers of law meet philosophers of technology (pp. 41–64). Abingdon: Routledge.Google Scholar
  70. Veale, M., & Edwards, L. (2018). Clarity, surprises, and further questions in the Article 29 Working Party draft guidance on automated decision-making and profiling. Computer Law and Security Review, 34, 398–404.CrossRefGoogle Scholar
  71. Wachter, S., Mittelstadt, B. D., & Floridi, L. (2017a). Transparent, explainable, and accountable AI for robotics. Science Robotics, 2(6).Google Scholar
  72. Wachter, S., Mittelstadt, B. D., & Floridi, L. (2017b). Why a right to explanation of automated decision-making does not exist in the General Data Protection Regulation. International Data Privacy Law, 7(2), 76–99.CrossRefGoogle Scholar
  73. Waldron, J. (1990). The law. London: Routledge.CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  1. 1.Department of PhilosophyUniversity of OtagoDunedinNew Zealand
  2. 2.Department of Computer ScienceUniversity of OtagoDunedinNew Zealand
  3. 3.Faculty of LawUniversity of OtagoDunedinNew Zealand

Personalised recommendations