Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

The role of human in the loop: lessons from D3R challenge 4


The rapid development of new machine learning techniques led to significant progress in the area of computer-aided drug design. However, despite the enormous predictive power of new methods, they lack explainability and are often used as black boxes. The most important decisions in drug discovery are still made by human experts who rely on intuitions and simplified representation of the field. We used D3R Grand Challenge 4 to model contributions of human experts during the prediction of the structure of protein–ligand complexes, and prediction of binding affinities for series of ligands in the context of absence or abundance of experimental data. We demonstrated that human decisions have a series of biases: a tendency to focus on easily identifiable protein–ligand interactions such as hydrogen bonds, and neglect for a more distributed and complex electrostatic interactions and solvation effects. While these biases still allow human experts to compete with blind algorithms in some areas, the underutilization of the information leads to significantly worse performance in data-rich tasks such as binding affinity prediction.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7


  1. 1.

    Sunseri J, King JE, Francoeur PG, Koes DR (2019) J Comput Aided Mol Des 33(1):19

  2. 2.

    Zhong F, Xing J, Li X, Liu X, Fu Z, Xiong Z, Lu D, Wu X, Zhao J, Tan X, Li F, Luo X, Li Z, Chen K, Zheng M, Jiang H (2018) Sci China Life Sci 61(10):1191

  3. 3.

    Zhang L, Tan J, Han D, Zhu H (2017) Drug Discov Today 22(11):1680

  4. 4.

    Macalino SJ, Gosu V, Hong S, Choi S (2015) Arch Pharm Res 38(9):1686

  5. 5.

    Baig MH, Ahmad K, Rabbani G, Danishuddin M, Choi I (2018) Curr Neuropharmacol 16(6):740

  6. 6.

    (2018) Nat Biomed Eng 2:709

  7. 7.

    Holzinger A, Biemann C, Pattichis CS, Kell DB (2017) What do we need to build explainable AI systems for the medical domain? arXiv preprint arXiv

  8. 8.

    Vassar R, Kovacs DM, Yan R, Wong PC (2009) J Neurosci 29(41):12787

  9. 9.

    Kitchen DB, Decornez H, Furr JR, Bajorath J (2004) Nat Rev Drug Discov 3(11):935

  10. 10.

    Liu J, Wang R (2015) J Chem Inf Model 55(3):475

  11. 11.

    Guedes IA, Pereira FSS, Dardenne LE (2018) Front Pharmacol 9:1089

  12. 12.

    Wilkinson RD, Williams R, Scott CJ, Burden RE (2015) Biol Chem 396(8):867

  13. 13.

    Ameriks MK, Bembenek SD, Burdett MT, Choong IC, Edwards JP, Gebauer D, Gu Y, Karlsson L, Purkey HE, Staker BL, Sun S, Thurmond RL, Zhu J (2010) Bioorg Med Chem Lett 20(14):4060

  14. 14.

    Wiener DK, Lee-Dutra A, Bembenek S, Nguyen S, Thurmond RL, Sun S, Karlsson L, Grice CA, Jones TK, Edwards JP (2010) Bioorg Med Chem Lett 20(7):2379

  15. 15.

    Ameriks MK, Axe FU, Bembenek SD, Edwards JP, Gu Y, Karlsson L, Randal M, Sun S, Thurmond RL, Zhu J (2009) Bioorg Med Chem Lett 19(21):6131

  16. 16.

    Levin NMB, Pintro VO, Bitencourt-Ferreira G, de Mattos BB, de Castro Silverio A, de Azevedo WF Jr (2018) Biophys Chem 235:1

  17. 17.

    Zhang L, Ai HX, Li SM, Qi MY, Zhao J, Zhao Q, Liu HS (2017) Oncotarget 8(47):83142

  18. 18.

    Park H, Eom JW, Kim YH (2014) J Chem Inf Model 54(7):2139

  19. 19.

    O'Boyle NM, Banck M, James CA, Morley C, Vandermeersch T, Hutchison GR (2011) J Cheminform 3(1):33

  20. 20.

    Cheeseright T, Mackey M, Rose S, Vinter A (2006) J Chem Inf Model 46(2):665

  21. 21.

    Stroganov OV, Novikov FN, Zeifman AA, Stroylov VS, Chilov GG (2011) Proteins 79(9):2693

  22. 22.

    Stroganov OV, Novikov FN, Stroylov VS, Kulkov V, Chilov GG (2008) J Chem Inf Model 48(12):2371

  23. 23.

    Ester M, Kriegel H-P, #246 S, Xu X. A density-based algorithm for discovering clusters a density-based algorithm for discovering clusters in large spatial databases with noise. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining. Portland, Oregon: AAAI Press, 1996:226

  24. 24.

    Novikov FN, Stroylov VS, Stroganov OV, Chilov GG (2010) J Mol Model 16(7):1223

  25. 25.

    Machauer R, Laumen K, Veenstra S, Rondeau JM, Tintelnot-Blomley M, Betschart C, Jaton AL, Desrayaud S, Staufenbiel M, Rabe S, Paganetti P, Neumann U (2009) Bioorg Med Chem Lett 19(5):1366

  26. 26.

    Gaulton A, Bellis LJ, Bento AP, Chambers J, Davies M, Hersey A, Light Y, McGlinchey S, Michalovich D, Al-Lazikani B, Overington JP (2012) Nucleic Acids Res 40:D1100

  27. 27.

    Gaieb Z, Parks CD, Chiu M, Yang H, Shao C, Walters WP, Lambert MH, Nevins N, Bembenek SD, Ameriks MK, Mirzadegan T, Burley SK, Amaro RE, Gilson MK (2019) J Comput Aided Mol Des 33(1):1

  28. 28.

    Team RC. R: A language and environment for statistical computing.

  29. 29.

    The team. H2O: Scalable machine learning (2015),

  30. 30.

    The team. h2o: R Interface for H2O (2015),

  31. 31.

    Wolpert DH (1992) Neural Netw 5(2):241–259

  32. 32.

    Breiman L (2001) Mach Learn 45(1):5

  33. 33.

    Friedman JH (2001) Annals of statistics:1189

  34. 34.

    LeCun Y, Bengio Y, Hinton G (2015) Nature 521(7553):436

  35. 35.

    Carlson HA (2016) J Chem Inf Model 56(6):951

  36. 36.

    Taylor R, Kennard O, Versichel W (1984) Acta Cryst B40:280–288

Download references

Author information

Correspondence to Oleg V. Stroganov.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Stroganov, O.V., Novikov, F.N., Medvedev, M.G. et al. The role of human in the loop: lessons from D3R challenge 4. J Comput Aided Mol Des 34, 121–130 (2020).

Download citation


  • Lead finder
  • D3R
  • Drug design data resource
  • Molecular docking
  • Human in the loop
  • Machine learning