Journal of Digital Imaging

, Volume 32, Issue 5, pp 693–701 | Cite as

Prior to Initiation of Chemotherapy, Can We Predict Breast Tumor Response? Deep Learning Convolutional Neural Networks Approach Using a Breast MRI Tumor Dataset

  • Richard HaEmail author
  • Christine Chin
  • Jenika Karcich
  • Michael Z. Liu
  • Peter Chang
  • Simukayi Mutasa
  • Eduardo Pascual Van Sant
  • Ralph T. Wynn
  • Eileen Connolly
  • Sachin Jambawalikar


We hypothesize that convolutional neural networks (CNN) can be used to predict neoadjuvant chemotherapy (NAC) response using a breast MRI tumor dataset prior to initiation of chemotherapy. An institutional review board-approved retrospective review of our database from January 2009 to June 2016 identified 141 locally advanced breast cancer patients who (1) underwent breast MRI prior to the initiation of NAC, (2) successfully completed adriamycin/taxane-based NAC, and (3) underwent surgical resection with available final surgical pathology data. Patients were classified into three groups based on their NAC response confirmed on final surgical pathology: complete (group 1), partial (group 2), and no response/progression (group 3). A total of 3107 volumetric slices of 141 tumors were evaluated. Breast tumor was identified on first T1 postcontrast dynamic images and underwent 3D segmentation. CNN consisted of ten convolutional layers, four max-pooling layers, and dropout of 50% after a fully connected layer. Dropout, augmentation, and L2 regularization were implemented to prevent overfitting of data. Non-linear functions were modeled by a rectified linear unit (ReLU). Batch normalization was used between the convolutional and ReLU layers to limit drift of layer activations during training. A three-class neoadjuvant prediction model was evaluated (group 1, group 2, or group 3). The CNN achieved an overall accuracy of 88% in three-class prediction of neoadjuvant treatment response. Three-class prediction discriminating one group from the other two was analyzed. Group 1 had a specificity of 95.1% ± 3.1%, sensitivity of 73.9% ± 4.5%, and accuracy of 87.7% ± 0.6%. Group 2 (partial response) had a specificity of 91.6% ± 1.3%, sensitivity of 82.4% ± 2.7%, and accuracy of 87.7% ± 0.6%. Group 3 (no response/progression) had a specificity of 93.4% ± 2.9%, sensitivity of 76.8% ± 5.7%, and accuracy of 87.8% ± 0.6%. It is feasible for current deep CNN architectures to be trained to predict NAC treatment response using a breast MRI dataset obtained prior to initiation of chemotherapy. Larger dataset will likely improve our prediction model.


Breast MRI Chemotherapy treatment response Convolutional neural network 


Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.


  1. 1.
    Rastogi P, Anderson SJ, Bear HD, Geyer CE, Kahlenberg MS, Robidoux A, Margolese RG, Hoehn JL, Vogel VG, Dakhil SR, Tamkus D, King KM, Pajon ER, Wright MJ, Robert J, Paik S, Mamounas EP, Wolmark N: Preoperative chemotherapy: updates of National Surgical Adjuvant Breast and Bowel Project Protocols B-18 and B-27. J Clin Oncol 26:778–785, 2008CrossRefGoogle Scholar
  2. 2.
    Gianni L, Pienkowski T, Im YH, Tseng LM, Liu MC, Lluch A, Starosławska E, de la Haba-Rodriguez J, Im SA, Pedrini JL, Poirier B, Morandi P, Semiglazov V, Srimuninnimit V, Bianchi GV, Magazzù D, McNally V, Douthwaite H, Ross G, Valagussa P: 5-year analysis of neoadjuvant pertuzumab and trastuzumab in patients with locally advanced, inflammatory, or early-stage HER2-positive breast cancer (NeoSphere): a multicentre, open-label, phase 2 randomised trial. Lancet Oncol 17:791–800, 2016CrossRefGoogle Scholar
  3. 3.
    Cortazar P, Zhang L, Untch M, Mehta K, Costantino JP, Wolmark N, Bonnefoi H, Cameron D, Gianni L, Valagussa P, Swain SM, Prowell T, Loibl S, Wickerham DL, Bogaerts J, Baselga J, Perou C, Blumenthal G, Blohmer J, Mamounas EP, Bergh J, Semiglazov V, Justice R, Eidtmann H, Paik S, Piccart M, Sridhara R, Fasching PA, Slaets L, Tang S, Gerber B, Geyer, Jr CE, Pazdur R, Ditsch N, Rastogi P, Eiermann W, von Minckwitz G: Pathological complete response and long-term clinical benefit in breast cancer: the CTNeoBC pooled analysis. Lancet 384:164–172, 2014CrossRefGoogle Scholar
  4. 4.
    Esserman LJ, Berry DA, DeMichele A, Carey L, Davis SE, Buxton M, Hudis C, Gray JW, Perou C, Yau C, Livasy C, Krontiras H, Montgomery L, Tripathy D, Lehman C, Liu MC, Olopade OI, Rugo HS, Carpenter JT, Dressler L, Chhieng D, Singh B, Mies C, Rabban J, Chen YY, Giri D, van 't Veer L, Hylton N: Pathologic complete response predicts recurrence-free survival more effectively by cancer subset: results from the I-SPY 1 TRIAL--CALGB 150007/150012, ACRIN 6657. J Clin Oncol 30:3242–3249, 2012CrossRefGoogle Scholar
  5. 5.
    von Minckwitz G, Untch M, Blohmer JU, Costa SD, Eidtmann H, Fasching PA, Gerber B, Eiermann W, Hilfrich J, Huober J, Jackisch C, Kaufmann M, Konecny GE, Denkert C, Nekljudova V, Mehta K, Loibl S: Definition and impact of pathologic complete response on prognosis after neoadjuvant chemotherapy in various intrinsic breast cancer subtypes. J Clin Oncol 30:1796–1804, 2012CrossRefGoogle Scholar
  6. 6.
    Wang-Lopez Q, Chalabi N, Abrial C, Radosevic-Robin N, Durando X, Mouret-Reynier MA, Benmammar KE, Kullab S, Bahadoor M, Chollet P, Penault-Llorca F, Nabholtz JM: Can pathologic complete response (pCR) be used as a surrogate marker of survival after neoadjuvant therapy for breast cancer? Crit Rev Oncol Hematol 95:88–104, 2015CrossRefGoogle Scholar
  7. 7.
    Broglio KR, Quintana M, Foster M, Olinger M, McGlothlin A, Berry SM, Boileau JF, Brezden-Masley C, Chia S, Dent S, Gelmon K, Paterson A, Rayson D, Berry DA: Association of pathologic complete response to neoadjuvant therapy in HER2-positive breast cancer with long-term outcomes: a meta-analysis. JAMA Oncol 2:751–760, 2016CrossRefGoogle Scholar
  8. 8.
    Carey LA: Neoadjuvant clinical trial designs: challenges of the genomic era. Breast 24(Suppl 2):S88–S90, 2015CrossRefGoogle Scholar
  9. 9.
    Carey LA, Winer EP: I-SPY 2—toward more rapid progress in breast cancer treatment. N Engl J Med 375:83–84, 2016CrossRefGoogle Scholar
  10. 10.
    Park JW, Liu MC, Yee D, Yau C, van 't Veer L, Symmans WF, Paoloni M, Perlmutter J, Hylton NM, Hogarth M, DeMichele A, Buxton MB, Chien AJ, Wallace AM, Boughey JC, Haddad TC, Chui SY, Kemmer KA, Kaplan HG, Isaacs C, Nanda R, Tripathy D, Albain KS, Edmiston KK, Elias AD, Northfelt DW, Pusztai L, Moulder SL, Lang JE, Viscusi RK, Euhus DM, Haley BB, Khan QJ, Wood WC, Melisko M, Schwab R, Helsten T, Lyandres J, Davis SE, Hirst GL, Sanil A, Esserman LJ, Berry DA, I-SPY 2 Investigators: Adaptive randomization of neratinib in early breast cancer. N Engl J Med 375:11–22, 2016CrossRefGoogle Scholar
  11. 11.
    Rugo HS, Olopade OI, DeMichele A, Yau C, van 't Veer L, Buxton MB, Hogarth M, Hylton NM, Paoloni M, Perlmutter J, Symmans WF, Yee D, Chien AJ, Wallace AM, Kaplan HG, Boughey JC, Haddad TC, Albain KS, Liu MC, Isaacs C, Khan QJ, Lang JE, Viscusi RK, Pusztai L, Moulder SL, Chui SY, Kemmer KA, Elias AD, Edmiston KK, Euhus DM, Haley BB, Nanda R, Northfelt DW, Tripathy D, Wood WC, Ewing C, Schwab R, Lyandres J, Davis SE, Hirst GL, Sanil A, Berry DA, Esserman LJ, I-SPY 2 Investigators: Adaptive randomization of veliparib-carboplatin treatment in breast cancer. N Engl J Med 375:23–34, 2016CrossRefGoogle Scholar
  12. 12.
    von Minckwitz G, Blohmer JU, Costa SD, Denkert C, Eidtmann H, Eiermann W, Gerber B, Hanusch C, Hilfrich J, Huober J, Jackisch C, Kaufmann M, Kümmel S, Paepke S, Schneeweiss A, Untch M, Zahm DM, Mehta K, Loibl S: Response-guided neoadjuvant chemotherapy for breast cancer. J Clin Oncol 31:3623–3630, 2013CrossRefGoogle Scholar
  13. 13.
    Gu YL, Pan SM, Ren J, Yang ZX, Jiang GQ: Role of magnetic resonance imaging in detection of pathologic complete remission in breast cancer patients treated with neoadjuvant chemotherapy: a meta-analysis. Clin Breast Cancer 17:245–255, 2017CrossRefGoogle Scholar
  14. 14.
    Li W, Arasu V, Newitt DC, Jones EF, Wilmes L, Gibbs J, Kornak J, Joe BN, Esserman LJ, Hylton NM, ACRIN 6657 Trial Team and I-SPY Investigators Network: Effect of MR imaging contrast thresholds on prediction of neoadjuvant chemotherapy response in breast cancer subtypes: a subgroup analysis of the ACRIN 6657/I-SPY 1 TRIAL. Tomography 2:378–387, 2016CrossRefGoogle Scholar
  15. 15.
    Hylton NM, Blume JD, Bernreuter WK, Pisano ED, Rosen MA, Morris EA, Weatherall PT, Lehman CD, Newstead GM, Polin S, Marques HS, Esserman LJ, Schnall MD, ACRIN 6657 Trial Team and I-SPY 1 TRIAL Investigators: Locally advanced breast cancer: MR imaging for prediction of response to neoadjuvant chemotherapy—results from ACRIN 6657/I-SPY TRIAL. Radiology 263:663–672, 2012CrossRefGoogle Scholar
  16. 16.
    Hylton NM, Gatsonis CA, Rosen MA, Lehman CD, Newitt DC, Partridge SC, Bernreuter WK, Pisano ED, Morris EA, Weatherall PT, Polin SM, Newstead GM, Marques HS, Esserman LJ, Schnall MD, For the ACRIN 6657 Trial Team and I-SPY 1 TRIAL Investigators: Neoadjuvant chemotherapy for breast cancer: functional tumor volume by MR imaging predicts recurrence-free survival-results from the ACRIN 6657/CALGB 150007 I-SPY 1 TRIAL. Radiology 279:44–55, 2016CrossRefGoogle Scholar
  17. 17.
    Weis JA, Miga MI, Yankeelov TE: Three-dimensional image-based mechanical modeling for predicting the response of breast cancer to neoadjuvant therapy. Comput Methods Appl Mech Eng 314:494–512, 2017CrossRefGoogle Scholar
  18. 18.
    Yankeelov TE: Integrating imaging data into predictive biomathematical and biophysical models of cancer. ISRN Biomath 2012:1–12, 2012CrossRefGoogle Scholar
  19. 19.
    LeCun Y, Bengio Y, Hinton G: Deep learning. Nature 521:436–444, 2015CrossRefGoogle Scholar
  20. 20.
    Hammond ME, Hayes DF, Dowsett M et al.: American Society of Clinical Oncology/College of American Pathologists guideline recommendations for immunohistochemical testing of estrogen and progesterone receptors in breast cancer. J Clin Oncol 28:2784–2795, 2010CrossRefGoogle Scholar
  21. 21.
    Wolff AC, Hammond ME, Hicks DG, Dowsett M, McShane L, Allison KH, Allred DC, Bartlett JM, Bilous M, Fitzgibbons P, Hanna W, Jenkins RB, Mangu PB, Paik S, Perez EA, Press MF, Spears PA, Vance GH, Viale G, Hayes DF, American Society of Clinical Oncology, College of American Pathologists: Recommendations for human epidermal growth factor receptor 2 testing in breast cancer: American Society of Clinical Oncology/College of American Pathologists clinical practice guideline update. Arch Pathol Lab Med 138:241–256, 2014CrossRefGoogle Scholar
  22. 22.
    Pieper S, Halle M, Kikinis R. 3D Slicer. In: 2004 2nd IEEE International Symposium on Biomedical Imaging: Macro to Nano (IEEE Cat No 04EX821). IEEE; p. 632–635.Google Scholar
  23. 23.
    Kingma DP, Ba J. Adam: a method for stochastic optimization. arXiv:1412.6980Google Scholar
  24. 24.
    Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R: Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 15:1929–1958, 2014Google Scholar
  25. 25.
    Therasse P, Arbuck SG, Eisenhauer EA, Wanders J, Kaplan RS, Rubinstein L, Verweij J, van Glabbeke M, van Oosterom AT, Christian MC, Gwyther SG: New guidelines to evaluate the response to treatment in solid tumors. J Natl Cancer Inst 92:205–216, 2000CrossRefGoogle Scholar
  26. 26.
    Li X, Arlinghaus LR, Ayers GD, Chakravarthy AB, Abramson RG, Abramson VG, Atuegwu N, Farley J, Mayer IA, Kelley MC, Meszoely IM, Means-Powell J, Grau AM, Sanders M, Bhave SR, Yankeelov TE: DCE-MRI analysis methods for predicting the response of breast cancer to neoadjuvant chemotherapy: pilot study findings. Magn Reson Med 71:1592–1602, 2014CrossRefGoogle Scholar
  27. 27.
    Li X, Abramson RG, Arlinghaus LR, Kang H, Chakravarthy AB, Abramson VG, Farley J, Mayer IA, Kelley MC, Meszoely IM, Means-Powell J, Grau AM, Sanders M, Yankeelov TE: Multiparametric magnetic resonance imaging for predicting pathological response after the first cycle of neoadjuvant chemotherapy in breast cancer. Invest Radiol 50:195–204, 2015CrossRefGoogle Scholar
  28. 28.
    Ah-See ML, Makris A, Taylor NJ et al.: Early changes in functional dynamic magnetic resonance imaging predict for pathologic response to neoadjuvant chemotherapy in primary breast cancer. Clin Cancer Res 14:6580–6589, 2008CrossRefGoogle Scholar
  29. 29.
    Atuegwu NC, Arlinghaus LR, Li X, Chakravarthy AB, Abramson VG, Sanders ME, Yankeelov TE: Parameterizing the logistic model of tumor growth by DW-MRI and DCE-MRI data to predict treatment response and changes in breast cancer cellularity during neoadjuvant chemotherapy. Transl Oncol 6:256–264, 2013CrossRefGoogle Scholar
  30. 30.
    Weis JA, Miga MI, Arlinghaus LR, Li X, Abramson V, Chakravarthy AB, Pendyala P, Yankeelov TE: Predicting the response of breast cancer to neoadjuvant therapy using a mechanically coupled reaction-diffusion model. Cancer Res 75:4697–4707, 2015CrossRefGoogle Scholar
  31. 31.
    Ravichandran K, Braman N, Janowczyk A, Madabushi A: A deep learning classifier for prediction of pathological complete response to neoadjuvant chemotherapy from baseline breast DCE-MRI. SPIE medical imaging 2018: computer-aided diagnosis 105750C, 2018Google Scholar
  32. 32.
    Houssami N, Macaskill P, von Minckwitz G, Marinovich ML, Mamounas E: Meta-analysis of the association of breast cancer subtype and pathologic complete response to neoadjuvant chemotherapy. Eur J Cancer 48:3342–3352, 2012CrossRefGoogle Scholar
  33. 33.
    Sorlie T, Perou CM, Tibshirani R, Aas T, Geisler S, Johnsen H, Hastie T, Eisen MB, van de Rijn M, Jeffrey SS, Thorsen T, Quist H, Matese JC, Brown PO, Botstein D, Lonning PE, Borresen-Dale AL: Gene expression patterns of breast carcinomas distinguish tumor subclasses with clinical implications. Proc Natl Acad Sci U S A 98:10869–10874, 2001CrossRefGoogle Scholar
  34. 34.
    Carey LA, Perou CM, Livasy CA, Dressler LG, Cowan D, Conway K, Karaca G, Troester MA, Tse CK, Edmiston S, Deming SL, Geradts J, Cheang MCU, Nielsen TO, Moorman PG, Earp HS, Millikan RC: Race, breast cancer subtypes, and survival in the Carolina Breast Cancer Study. JAMA 295:2492–2502, 2006CrossRefGoogle Scholar
  35. 35.
    Nguyen PL, Taghian AG, Katz MS, Niemierko A, Abi Raad RF, Boon WL, Bellon JR, Wong JS, Smith BL, Harris JR: Breast cancer subtype approximated by estrogen receptor, progesterone receptor, and HER-2 is associated with local and distant recurrence after breast-conserving therapy. J Clin Oncol. 26(14):2373–2378, 2008CrossRefGoogle Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2018

Authors and Affiliations

  • Richard Ha
    • 1
    Email author
  • Christine Chin
    • 2
  • Jenika Karcich
    • 1
  • Michael Z. Liu
    • 3
  • Peter Chang
    • 4
  • Simukayi Mutasa
    • 1
  • Eduardo Pascual Van Sant
    • 1
  • Ralph T. Wynn
    • 1
  • Eileen Connolly
    • 2
  • Sachin Jambawalikar
    • 3
  1. 1.Department of RadiologyColumbia University Irving Medical CenterNew YorkUSA
  2. 2.Division of Radiation OncologyColumbia University Medical CenterNew YorkUSA
  3. 3.Department of Medical PhysicsColumbia University Medical CenterNew YorkUSA
  4. 4.Department of RadiologyUC San Francisco Medical CenterSan FranciscoUSA

Personalised recommendations