FABnet: feature attention-based network for simultaneous segmentation of microvessels and nerves in routine histology images of oral cancer

  • M. M. FrazEmail author
  • S. A. Khurram
  • S. Graham
  • M. Shaban
  • M. Hassan
  • A. Loya
  • N. M. Rajpoot
Original Article


Perineural invasion (PNI), lymphovascular invasion (LVI) and tumor angiogenesis have strong correlation with cancer recurrence, metastasis and poor patient survival. The accurate segmentations of nerves and microvessels can be considered as the preliminary step in objective identification of PNI, LVI and tumor angiogenic analysis in histology images. We proposed a deep network for simultaneous segmentation of microvessel and nerves in routinely used H&E-stained histology images. The network is designed as an encoder–decoder architecture with embedded feature attention blocks and an uncertainty prediction. The proposed network uses Xception residual blocks, followed by atrous spatial pyramid pooling for feature extraction at multiple scales. Feature attention blocks are used in the skip connections from encoder to decoder as well as in the decoder up-sampling, which enables the network in focusing on more salient features while making prediction for segmentation. The method is evaluated using 7780 images of size 512 × 512 pixels, extracted from 20 WSIs of oral squamous cell carcinoma tissue at 20× magnification. The ensemble of network outputs at test time is used to obtain a better segmentation result and simultaneous generation of network prediction uncertainty map. The proposed network achieves state-of-the-art results compared to currently used deep neural networks for semantic segmentation (FCN-8, U-Net, Segnet and DeepLabV3+). The proposed network also gives robust segmentation performance when applied to the full digital whole slide image.


Computational pathology H&E-stained images Tumor microenvironment analysis Deep neural network Feature attention Uncertainty prediction 


Compliance with ethical standards

Conflict of interest

The authors declared that they have no conflict of interest.


  1. 1.
    Nishida N, Yano H, Nishida T, Kamura T, Kojiro M (2006) Angiogenesis in cancer. Vasc Health Risk Manag 2(3):213CrossRefGoogle Scholar
  2. 2.
    Liebig C, Ayala G, Wilks JA, Berger DH, Albo D (2009) Perineural invasion in cancer: a review of the literature. Cancer Interdiscip Int J Am Cancer Soc 115(15):3379Google Scholar
  3. 3.
    Al-Sukhni E, Attwood K, Gabriel EM, LeVea CM, Kanehira K, Nurkin SJ (2017) Lymphovascular and perineural invasion are associated with poor prognostic features and outcomes in colorectal cancer: a retrospective cohort study. Int J Surg 37:42. CrossRefGoogle Scholar
  4. 4.
    Noma D, Inamura K, Matsuura Y, Hirata Y, Nakajima T, Yamazaki H, Hirai Y, Ichinose J, Nakao M, Ninomiya H, Mun M, Nakagawa K, Masuda M, Ishikawa Y, Okumura S (2018) Prognostic effect of lymphovascular invasion on TNM staging in stage I non-small-cell lung cancer. Clinical Lung Cancer 19(1):e109. CrossRefGoogle Scholar
  5. 5.
    Liebig C, Ayala G, Wilks J, Verstovsek G, Liu H, Agarwal N, Berger DH, Albo D (2009) Perineural invasion is an independent predictor of outcome in colorectal cancer. J Clin Oncol 27(31):5131CrossRefGoogle Scholar
  6. 6.
    Kurtz KA, Hoffman HT, Zimmerman MB, Robinson RA (2005) Perineural and vascular invasion in oral cavity squamous carcinoma: increased incidence on re-review of slides and by using immunohistochemical enhancement. Arch Pathol Lab Med 129(3):354Google Scholar
  7. 7.
    Leon SP, Folkerth RD, Black PM (1996) Microvessel density is a prognostic indicator for patients with astroglial brain tumors. Cancer Interdiscip Int J Am Cancer Soc 77(2):362Google Scholar
  8. 8.
    Long J, Shelhamer E, Darrell T. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440Google Scholar
  9. 9.
    Ronneberger O, Fischer P, Brox T. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241Google Scholar
  10. 10.
    Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell 39(12):2481CrossRefGoogle Scholar
  11. 11.
    Chen LC, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Computer vision—ECCV. Springer, NewYork, pp 833–851Google Scholar
  12. 12.
    Janowczyk A, Madabhushi A (2016) Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. J Pathol Inform 7(1):29CrossRefGoogle Scholar
  13. 13.
    Jin Z, Zhou G, Gao D, Zhang Y (2018) EEG classification using sparse Bayesian extreme learning machine for brain–computer interface. Neural Comput Appl. CrossRefGoogle Scholar
  14. 14.
    Sirinukunwattana K, Raza SEA, Tsang YW, Snead DRJ, Cree IA, Rajpoot NM (2016) Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans Med Imaging 35(5):1196CrossRefGoogle Scholar
  15. 15.
    Xu J, Xiang L, Liu Q, Gilmore H, Wu J, Tang J, Madabhushi A (2016) Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images. IEEE Trans Med Imaging 35(1):119CrossRefGoogle Scholar
  16. 16.
    Saha M, Chakraborty C, Racoceanu D (2018) Efficient deep learning model for mitosis detection using breast histopathology images. Comput Med Imaging Graph 64:29CrossRefGoogle Scholar
  17. 17.
    Öztürk Ş, Akdemir B (2019) A convolutional neural network model for semantic segmentation of mitotic events in microscopy images. Neural Comput Appl 31:3719–3728CrossRefGoogle Scholar
  18. 18.
    Saltz J, Gupta R, Hou L, Thorsson V (2018) Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Rep. 23(1):181. CrossRefGoogle Scholar
  19. 19.
    Xu J, Luo X, Wang G, Gilmore H, Madabhushi A (2016) Adeep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing 191:214CrossRefGoogle Scholar
  20. 20.
    Wang X, Guo Y, Wang Y, Yu J (2019) Automatic breast tumor detection in ABVS images based on convolutional neural network and superpixel patterns. Neural Comput Appl 31(4):1069–1081CrossRefGoogle Scholar
  21. 21.
    Graham S, Chen H, Gamper J, Dou Q, Heng PA, Snead D, Tsang YW, Rajpoot N (2018) MILD-Net: minimal information loss dilated network for gland instance segmentation in colon histology images. Med Image Anal 52:199–211CrossRefGoogle Scholar
  22. 22.
    Chen H, Qi X, Yu L, Dou Q, Qin J, Heng PA (2017) DCAN: deep contour-aware networks for object instance segmentation from histology images. Med Image Anal 36:135. CrossRefGoogle Scholar
  23. 23.
    Xu Y, Li Y, Wang Y, Liu M, Fan Y, Lai M, Chang EI (2017) Gland instance segmentation using deep multichannel neural networks. IEEE Trans Biomed Eng 64(12):2901. CrossRefGoogle Scholar
  24. 24.
    Selvanambi R, Natarajan J, Karuppiah M, Islam SH, Hassan MM, Fortino G (2018) Lung cancer prediction using higher-order recurrent neural network based on glowworm swarm optimization. Neural Comput Appl. CrossRefGoogle Scholar
  25. 25.
    Kather JN, Marx A, Reyes-Aldasoro CC, Schad LR, Zöllner FG, Weis CA (2015) Continuous representation of tumor microvessel density and detection of angiogenic hotspots in histological whole-slide images. Oncotarget 6(22):19163CrossRefGoogle Scholar
  26. 26.
    Yi F, Yang L, Wang S, Guo L, Huang C, Xie Y, Xiao G (2018) Microvessel prediction in H&E stained pathology images using fully convolutional neural networks. BMC Bioinform 19(1):64CrossRefGoogle Scholar
  27. 27.
    Fraz MM, Shaban M, Graham S, Khurram SA, Rajpoot NM (2018) Uncertainty driven pooling network for microvessel segmentation in routine histology images. In: Stoyanov D et al (eds) Computational pathology and ophthalmic medical image analysis. Springer, Cham, pp 156–164CrossRefGoogle Scholar
  28. 28.
    Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2018) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFS. IEEE Trans Pattern Anal Mach Intell 40(4):834CrossRefGoogle Scholar
  29. 29.
    Lin G, Milan A, Shen C, Reid I. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 5168–5177.
  30. 30.
    Romera E, Álvarez JM, Bergasa LM, Arroyo R (2018) ERFNet: efficient residual factorized ConvNet for real-time semantic segmentation. IEEE Trans Intell Transp Syst 19(1):263CrossRefGoogle Scholar
  31. 31.
    Mnih V, Heess N, Graves A et al (2014) Recurrent models of visual attention. In: Advances in neural information processing systems, pp 2204–2212Google Scholar
  32. 32.
    Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhudinov R, Zemel R, Bengio Y (2015) International conference on machine learning, pp 2048–2057Google Scholar
  33. 33.
    Du W, Wang Y, Qiao Y (2018) Recurrent spatial-temporal attention network for action recognition in videos. IEEE Trans Image Process 27(3):1347. MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Chen L, Yang Y, Wang J, Xu W, Yuille AL. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 3640–3649.
  35. 35.
    Beck JL, Katafygiotis LS (1998) Updating models and their uncertainties I: bayesian statistical framework. J Eng Mech 124(4):455CrossRefGoogle Scholar
  36. 36.
    Kendall A, Badrinarayanan V, Cipolla R (2015). arXiv preprint arXiv:1511.02680
  37. 37.
    Gal Y, Ghahramani Z. In: International conference on machine learning, pp 1050–1059Google Scholar
  38. 38.
    Kendall A, Gal Y. In: Advances in neural information processing systems, pp 5580–5590Google Scholar
  39. 39.
    Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2014). arXiv preprint arXiv:1412.7062
  40. 40.
    Szegedy C, Ioffe S, Vanhoucke V, Alemi AA. In: AAAI, vol 4, p 12Google Scholar
  41. 41.
    Xie S, Girshick R, Dollár P, Tu Z, He K. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 5987–5995Google Scholar
  42. 42.
    Chollet F. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1800–1807.
  43. 43.
    Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017). arXiv preprint arXiv:1704.04861
  44. 44.
    Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211MathSciNetCrossRefGoogle Scholar
  45. 45.
    Goodfellow I, Bengio Y, Courville A, Bengio Y (2016) Deep learning, vol 1. MIT Press, CambridgezbMATHGoogle Scholar
  46. 46.
    Bahdanau D, Cho K, Bengio Y (2014). arXiv preprint arXiv:1409.0473
  47. 47.
    He K, Gkioxari G, Dollár P, Girshick R. In: IEEE international conference on computer vision (ICCV), pp 2980–2988.
  48. 48.
    BenTaieb A, Hamarneh G. Intravascular imaging and computer assisted stenting, and large-scale annotation of biomedical data and expert label synthesis. Springer, pp 155–163Google Scholar
  49. 49.
    Graham S, Rajpoot NM. In: IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp 590–594.

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of WarwickCoventryUK
  2. 2.School of Clinical DentistryUniversity of SheffieldSheffieldUK
  3. 3.Shaukat Khanum Memorial Cancer Hospital Research CentreLahorePakistan
  4. 4.National University of Sciences and Technology (NUST)IslamabadPakistan
  5. 5.The Alan Turing InstituteLondonUK

Personalised recommendations