Journal of Computer-Aided Molecular Design

, Volume 33, Issue 1, pp 1–18 | Cite as

D3R Grand Challenge 3: blind prediction of protein–ligand poses and affinity rankings

  • Zied Gaieb
  • Conor D. Parks
  • Michael Chiu
  • Huanwang Yang
  • Chenghua Shao
  • W. Patrick Walters
  • Millard H. Lambert
  • Neysa Nevins
  • Scott D. Bembenek
  • Michael K. Ameriks
  • Tara Mirzadegan
  • Stephen K. Burley
  • Rommie E. AmaroEmail author
  • Michael K. GilsonEmail author


The Drug Design Data Resource aims to test and advance the state of the art in protein–ligand modeling by holding community-wide blinded, prediction challenges. Here, we report on our third major round, Grand Challenge 3 (GC3). Held 2017–2018, GC3 centered on the protein Cathepsin S and the kinases VEGFR2, JAK2, p38-α, TIE2, and ABL1, and included both pose-prediction and affinity-ranking components. GC3 was structured much like the prior challenges GC2015 and GC2. First, Stage 1 tested pose prediction and affinity ranking methods; then all available crystal structures were released, and Stage 2 tested only affinity rankings, now in the context of the available structures. Unique to GC3 was the addition of a Stage 1b self-docking subchallenge, in which the protein coordinates from all of the cocrystal structures used in the cross-docking challenge were released, and participants were asked to predict the pose of CatS ligands using these newly released structures. We provide an overview of the outcomes and discuss insights into trends and best-practices.


D3R Drug Design Data Resource Docking Scoring Ligand ranking Blinded prediction challenge 



This work was supported by National Institutes of Health (NIH) Grant No. 1U01GM111528 for the Drug Design Data Resource (D3R). We also thank OpenEye Scientific Software for generously donating the use of their software. RCSB Protein Data Bank is supported by NSF, NCI, NIGMS, and DOE (Grant No. NSF DBI-1338415). The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. MKG has an equity interest in, and is a co-founder and scientific advisor of, VeraChem LLC; REA has equity interest in and is a co-founder and scientific advisor of Actavalon, Inc.; and PW has an equity interest in Relay Pharmaceuticals, Inc. We also thank the reviewers for their helpful suggestions.

Supplementary material

10822_2018_180_MOESM1_ESM.docx (25 kb)
SI-description_submission (DOCX 24 KB)
10822_2018_180_MOESM2_ESM.docx (814 kb)
Supplementary_Figures_rv4 (DOCX 813 KB)
10822_2018_180_MOESM3_ESM.xlsx (30 kb)
Supplementary-Table1_experimental-data (XLSX 30 KB)
10822_2018_180_MOESM4_ESM.csv (15 kb)
Supplementary-Table2-CatS-properties (CSV 15 KB)
10822_2018_180_MOESM5_ESM.xlsx (23 kb)
Supplementary-Table3_KendallsTauFull (XLSX 22 KB)
10822_2018_180_MOESM6_ESM.csv (7 kb)
Supplementary-Table4-Stage1a-methods-stats (CSV 7 KB)
10822_2018_180_MOESM7_ESM.csv (8 kb)
Supplementary-Table5-Stage1b-methods-stats (CSV 7 KB)
10822_2018_180_MOESM8_ESM.csv (1 kb)
Supplementary-Table6-1a-1b-comp (CSV 1 KB)
10822_2018_180_MOESM9_ESM.xlsx (33 kb)
Supplementary-Table7_KendallsTau (XLSX 32 KB)
10822_2018_180_MOESM10_ESM.xlsx (22 kb)
Supplementary-Table8_MatthCorrCoeff (XLSX 22 KB)
10822_2018_180_MOESM11_ESM.xlsx (11 kb)
Supplementary-Table9_GC3_analysis_flips (XLSX 10 KB)
10822_2018_180_MOESM12_ESM.csv (0 kb)
Supplementary-Table10_max_pairs (CSV 0 KB)
10822_2018_180_MOESM13_ESM.xlsx (20 kb)
Supplementary-Table11_KendallsTau_CatS_XrayOnly (XLSX 20 KB) (247 kb)
SM_CatS_expt (ZIP 246 KB) (1.9 mb)
SM_KinaseData_DiscoverX (ZIP 1972 KB)


  1. 1.
    Macalino SJY, Gosu V, Hong S, Choi S (2015) Arch Pharm Res 38(9):1686–1701Google Scholar
  2. 2.
    Jorgensen WL (2004) Science 303(5665):1813–1818Google Scholar
  3. 3.
    Sliwoski G, Kothiwale S, Meiler J, LoweEW (2013) Pharmacol Rev 66(1):334–395Google Scholar
  4. 4.
    Irwin JJ, Shoichet BK (2016) J Med Chem 59(9):4103–4120Google Scholar
  5. 5.
    Amaro RE, Mulholland AJ (2018) Nat Rev Chem 2(4):148Google Scholar
  6. 6.
    Carlson HA (2016) J Chem Inf Model 56(6):951–954Google Scholar
  7. 7.
    Carlson HA, Smith RD, Damm-Ganamet KL, Stuckey JA, Ahmed A, Convery MA, Somers DO, Kranz M, Elkins PA, Cui G, Peishoff CE, Lambert MH, Dunbar JB (2016) J Chem Inf Model 56(6):1063–1077Google Scholar
  8. 8.
    Damm-Ganamet KL, Smith RD, Dunbar JB, Stuckey JA, Carlson HA (2013) J Chem Inf Model 53(8):1853–1870Google Scholar
  9. 9.
    Smith RD, Dunbar JB, Ung PM-U, Esposito EX, Yang C-Y, Wang S, Carlson HA (2011) J Chem Inf Model 51(9):2115–2131Google Scholar
  10. 10.
    Gaieb Z, Liu S, Gathiaka S, Chiu M, Yang H, Shao C, Feher VA, Walters WP, Kuhn B, Rudolph MG, Burley SK, Gilson MK, Amaro RE (2018) J Comput Aided Mol Des 32(1):1–20Google Scholar
  11. 11.
    Gathiaka S, Liu S, Chiu M, Yang H, Stuckey JA, Kang YN, Delproposto J, Kubish G, Dunbar JB, Carlson HA, Burley SK, Walters WP, Amaro RE, Feher VA, Gilson MK (2016) J Comput Aided Mol Des 30(9):651–668Google Scholar
  12. 12.
    Kontoyianni M, McClellan LM, Sokol GS (2004) J Med Chem 47(3):558–565Google Scholar
  13. 13.
    Kellenberger E, Rodrigo J, Muller P, Rognan D (2004) Proteins Struct Funct Bioinform 57(2):225–242Google Scholar
  14. 14.
    Cole JC, Murray CW, Nissink JWM, Taylor RD, Taylor R (2005) Proteins Struct Funct Bioinform 60(3):325–332Google Scholar
  15. 15.
    Huang S-Y, Grinter SZ, Zou X (2010) Phys Chem Chem Phys 12(40):12899Google Scholar
  16. 16.
    Hartshorn MJ, Verdonk ML, Chessari G, Brewerton SC, Mooij WTM, Mortenson PN, Murray CW (2007) J Med Chem 50(4):726–741Google Scholar
  17. 17.
    Leach AR, Shoichet BK, Peishoff CE (2006) J Med Chem 49(20):5851–5855Google Scholar
  18. 18.
    Thurmond RL, Sun S, Sehon CA, Baker SM, Cai H, Gu Y, Jiang W, Riley JP, Williams KN, Edwards JP, Karlsson L (2004) J Pharmacol Exp Ther 308(1):268–276Google Scholar
  19. 19.
    Drewry DH, Wells CI, Andrews DM, Angell R, Al-Ali H, Axtman AD, Capuzzi SJ, Elkins JM, Ettmayer P, Frederiksen M, Gileadi O, Gray N, Hooper A, Knapp S, Laufer S, Luecking U, Michaelides M, Müller S, Muratov E, Denny RA, Saikatendu KS, Treiber DK, Zuercher WJ, Willson TM (2017) PLoS ONE 12(8):e0181585Google Scholar
  20. 20.
    Dimova D, Bajorath J (2016) Mol Inform 35(5):181–191Google Scholar
  21. 21.
    Jacobson MP, Pincus DL, Rapp CS, Day TJF, Honig B, Shaw DE, Friesner RA (2004) Proteins Struct Funct Bioinform 55(2):351–367Google Scholar
  22. 22.
    Kendall MG (1938) Biometrika 30(1/2):81Google Scholar
  23. 23.
    Kendall MG (1945) Biometrika 33(3):239Google Scholar
  24. 24.
    Zwillinger D (2001) Standard probability and statistics tables and formulae, vol 43. CRC Press, Boca RatonGoogle Scholar
  25. 25.
    Gibbons J (2011) Nonparametric measures of association. SAGE Publications, Inc, Thousand Oaks, pp 17–29Google Scholar
  26. 26.
    Matthews BW (1975) Biochim Biophys Acta—Protein Struct 405(2):442–451Google Scholar
  27. 27.
    Wildman SA, Crippen GM (1999) J Chem Inf Comput Sci 39(5):868–873Google Scholar
  28. 28.
    Halgren TA, Murphy RB, Friesner RA, Beard HS, Frye LL, Pollard WT, Banks JL (2004) J Med Chem 47(7):1750–1759Google Scholar
  29. 29.
    Friesner RA, Banks JL, Murphy RB, Halgren TA, Klicic JJ, Mainz DT, Repasky MP, Knoll EH, Shelley M, Perry JK, Shaw DE, Francis P, Shenkin PS (2004) J Med Chem 47(7):1739–1749Google Scholar
  30. 30.
    Abagyan R, Totrov M, Kuznetsov D (1994) J Comput Chem 15(5):488–506Google Scholar
  31. 31.
    Stroganov OV, Novikov FN, Stroylov VS, Kulkov V, Chilov GG (2008) J Chem Inf Model 48(12):2371–2385Google Scholar
  32. 32.
    Kelley BP, Brown SP, Warren GL, Muchmore SW (2015) J Chem Inf Model 55(8):1771–1780Google Scholar
  33. 33.
    Koes DR, Baumgartner MP, Camacho CJ (2013) J Chem Inf Model 53(8):1893–1904Google Scholar
  34. 34.
    Zarbafian S, Moghadasi M, Roshandelpoor A, Nan F, Li K, Vakli P, Vajda S, Kozakov D, Paschalidis IC (2018) Sci Rep 8(1):5896Google Scholar
  35. 35.
    van Zundert GCP, Rodrigues JPGLM, Trellet M, Schmitz C, Kastritis PL, Karaca E, Melquiond ASJ, van Dijk M, de Vries SJ, Bonvin AMJJ (2016) J Mol Biol 428(4):720–725Google Scholar
  36. 36.
    Amaro RE, Baron R, McCammon JA (2008) J Comput Aided Mol Des 22(9):693–705Google Scholar
  37. 37.
    Korb O, Olsson TSG, Bowden SJ, Hall RJ, Verdonk ML, Liebeschuetz JW, Cole JC (2012) J Chem Inf Model 52(5):1262–1274Google Scholar
  38. 38.
    Amaro RE, Baudry J, Chodera J, Demir Ö, McCammon JA, Miao Y, Smith JC (2018) Biophys J 114(10):2271–2278Google Scholar
  39. 39.
    Tuccinardi T, Botta M, Giordano A, Martinelli A (2010) J Chem Inf Model 50(8):1432–1441Google Scholar
  40. 40.
    Kumar A, Zhang KYJ (2016) J Comput Aided Mol Des 30(6):457–469Google Scholar
  41. 41.
    Hawkins PCD, Skillman AG, Nicholls A (2007) J Med Chem 50(1):74–82Google Scholar
  42. 42.
    Cang Z, Mu L, Wei G-W (2018) PLOS Comput Biol 14(1):e1005929Google Scholar
  43. 43.
    Hochuli J, Helbling A, Skaist T, Ragoza M, Koes DR (2018) Mach Learn. arXiv:1803.02398Google Scholar
  44. 44.
    Warren GL, Andrews CW, Capelli A-M, Clarke B, LaLonde J, Lambert MH, Lindvall M, Nevins N, Semus SF, Senger S, Tedesco G, Wall ID, Woolven JM, Peishoff CE, Head MS (2006) J Med Chem 49(20):5912–5931Google Scholar
  45. 45.
    Shoichet BK, McGovern SL, Wei B, Irwin JJ (2002) Curr Opin Chem Biol 6(4):439–446Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Drug Design Data ResourceUniversity of California, San DiegoLa JollaUSA
  2. 2.RCSB Protein Data Bank, Institute for Quantitative Biomedicine, RutgersThe State University of New JerseyNew BrunswickUSA
  3. 3.Cancer Institute of New Jersey, RutgersThe State University of New JerseyNew BrunswickUSA
  4. 4.Relay TherapeuticsCambridgeUSA
  5. 5.GlaxoSmithKlineCollegevilleUSA
  6. 6.Janssen Research & DevelopmentSan DiegoUSA

Personalised recommendations