Causal inference for social discrimination reasoning

  • Bilal QureshiEmail author
  • Faisal Kamiran
  • Asim Karim
  • Salvatore Ruggieri
  • Dino Pedreschi


The discovery of discriminatory bias in human or automated decision making is a task of increasing importance and difficulty, exacerbated by the pervasive use of machine learning and data mining. Currently, discrimination discovery largely relies upon correlation analysis of decisions records, disregarding the impact of confounding biases. We present a method for causal discrimination discovery based on propensity score analysis, a statistical tool for filtering out the effect of confounding variables. We introduce causal measures of discrimination which quantify the effect of group membership on the decisions, and highlight causal discrimination/favoritism patterns by learning regression trees over the novel measures. We validate our approach on two real world datasets. Our proposed framework for causal discrimination has the potential to enhance the transparency of machine learning with tools for detecting discriminatory bias both in the training data and in the learning algorithms.


Social discrimination Fairness, accountability, and transparency Propensity score Causal analysis 


Compliance with Ethical Standards

Conflict of interests

The authors declare that they have no conflict of interest.


  1. Agresti, A. (2002). Categorical data analysis. Wiley series in probability and statistics, 2 edn. Wiley-Interscience.Google Scholar
  2. Austin, P.C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399–424.CrossRefGoogle Scholar
  3. Baeza-Yates, R.A. (2018). Bias on the web. Communications of the ACM, 61(6), 54–61.CrossRefGoogle Scholar
  4. Barocas, S., & Selbst, A.D. (2016). Big data’s disparate impact. California Law Review, 104.Google Scholar
  5. Bendic, M. (2007). Situation testing for employment discrimination in the United States of America. Horizons Stratégiques, 3(5), 17–39.Google Scholar
  6. Berk, R., Heidari, H., Jabbari, S., Kearns, M., Roth, A. (2018). Fairness in criminal justice risk assessments: the state of the art. Sociological Methods & Research.Google Scholar
  7. Bickel, P.J., Hammel, E.A., O’Connell, J.W. (1975). Sex bias in graduate admissions: data from Berkeley. Science, 187(4175), 398–404.CrossRefGoogle Scholar
  8. Bolukbasi, T., Chang, K., Zou, J.Y., Saligrama, V., Kalai, A.T. (2016). Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In NIPS (pp. 4349–4357).Google Scholar
  9. Bonchi, F., Hajian, S., Mishra, B., Ramazzotti, D. (2017). Exposing the probabilistic causal structure of discrimination. I. Journal Data Science and Analytics, 3(1), 1–21.CrossRefGoogle Scholar
  10. Breiman, L., Friedman, J, Olshen, R., Stone, C. (1984). Classification and regression trees. Wadsworth Publishing Company.Google Scholar
  11. Bryson, A., Dorsett, R., Purdon, S. (2002). The use of propensity score matching in the evaluation of active labour market policies. Crown.Google Scholar
  12. Calders, T., Karim, A., Kamiran, F., Ali, W., Zhang, X. (2013). Controlling attribute effect in linear regression. In ICDM (pp. 71–80): IEEE.Google Scholar
  13. Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys, 22(1), 31–72.CrossRefGoogle Scholar
  14. Dressel, J., & Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(1).Google Scholar
  15. Fortin, N., Lemieux, T., Firpo, S. (2011). Decomposition methods in economics. In Handbook of labor economics, (Vol. 4 pp. 1–102): Elsevier.Google Scholar
  16. Foster, S.R. (2004). Causation in antidiscrimination law: beyond intent versus impact. Houston Law Review, 41(5), 1469–1548.Google Scholar
  17. Grimes, D.A., & Schulz, K.F. (2002). Bias and causal associations in observational research. Lancet, 359, 248–252.CrossRefGoogle Scholar
  18. Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D. (2019). A survey of methods for explaining black box models. ACM Computing Survey, 51(5), 93:1–93:42.Google Scholar
  19. Guo, X.S., & Fraser, M.W. (2015). Propensity score analysis: statistical methods and applications, Sage Publications, Inc., 2.Google Scholar
  20. Kilbertus, N., Ball, P.J., Kusner, M.J., Weller, A., Silva, R. (2019). The sensitivity of counterfactual fairness to unmeasured confounding. In UAI (p. 213): AUAI Press.Google Scholar
  21. Kohavi, R., & Longbotham, R. (2017). Online controlled experiments and A/B testing. In Encyclopedia of machine learning and data mining (pp. 922–929): Springer.Google Scholar
  22. Kohler-Hausmann, I. (2019). Eddie Murphy and the dangers of counterfactual causal thinking about detecting racial discrimination . Northwestern University Law Rev, 113, 1163–1227.Google Scholar
  23. Kulshrestha, J., Eslami, M., Messias, J., Zafar, M.B., Ghosh, S., Gummadi, K.P., Karahalios, K. (2019). Search bias quantification: investigating political bias in social media and web search. Information Retrieval Journal, 22(1–2), 188–227.CrossRefGoogle Scholar
  24. Kusner, M.J., Loftus, J.R., Russell, C., Silva, R. (2017). Counterfactual fairness. In NIPS (pp. 4069–4079).Google Scholar
  25. Loftus, J.R., Russell, C., Kusner, M.J., Silva, R. (2018). Causal reasoning for algorithmic fairness. arXiv:abs/1805.05859.
  26. Luong, B.T., Ruggieri, S., Turini, F. (2011). k-NN as an implementation of situation testing for discrimination discovery and prevention. In KDD (pp. 502–510): ACM.Google Scholar
  27. Morgan, S.L., & Todd, J.L. (2008). A diagnostic routine for the detection of consequential heterogeneity of causal effects. Sociological Methodology, 38(1), 231–281.CrossRefGoogle Scholar
  28. Pearl, J. (2009). Causality: models, reasoning, and inference, 2nd edn. New York: Cambridge University Press.CrossRefGoogle Scholar
  29. Romei, A., & Ruggieri, S. (2014). A multidisciplinary survey on discrimination analysis. The Knowledge Engineering Review, 29(5), 582–638.CrossRefGoogle Scholar
  30. Rosenbaum, P.R., & Rubin, D.B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.MathSciNetCrossRefGoogle Scholar
  31. Shadish, W.R., Cook, T.D., Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton-Mifflin.Google Scholar
  32. Verma, S., & Rubin, J. (2018). Fairness definitions explained. In FairWare@ICSE (pp. 1–7): ACM.Google Scholar
  33. Wu, Y., Zhang, L., Wu, X. (2019). Counterfactual fairness: Unidentification, bound and algorithm. In IJCAI. (pp. 1438–1444).
  34. Zhang, J., & Bareinboim, E. (2018). Fairness in decision-making - the causal explanation formula. In AAAI: AAAI Press.Google Scholar
  35. Zhang, L., & Wu, X. (2017). Anti-discrimination learning: a causal modeling-based framework. I. Journal Data Science and Analytics, 4(1), 1–16.CrossRefGoogle Scholar
  36. Zhang, L., Wu, Y., Wu, X. (2016). Situation testing-based discrimination discovery: a causal inference approach. In IJCAI (pp. 2718–2724).Google Scholar
  37. Zhang, L., Wu, Y., Wu, X. (2017). Achieving non-discrimination in data release. In KDD (pp. 1335–1344): ACM.Google Scholar
  38. Zliobaite, I. (2017). Measuring discrimination in algorithmic decision making. Data Mining and Knowledge Discovery, 31(4), 1060–1089.MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Bilal Qureshi
    • 1
    Email author
  • Faisal Kamiran
    • 2
  • Asim Karim
    • 3
  • Salvatore Ruggieri
    • 4
    • 5
  • Dino Pedreschi
    • 4
    • 5
  1. 1.Addo.aiLahorePakistan
  2. 2.Information Technology University of the PunjabLahorePakistan
  3. 3.Lahore University of Management Sciences (LUMS)LahorePakistan
  4. 4.Università di PisaPisaItaly
  5. 5.ISTI-CNRPisaItaly

Personalised recommendations