Serial attention network for skin lesion segmentation

Abstract

Due to the influence of color, boundaries, and shapes of melanoma, the segmentation of the lesion area is still a challenging problem. In this paper, we propose a method to connect two attention modules (channel attention and spatial attention) serially and embed them into the skip connection of the encoder–decoder network. Different from previous work on image segmentation through attention mechanism, we have made a different combination of the existing popular spatial attention module and channel attention module. The experimental results show that the sequence combinations with channel attention in front and spatial attention in the rear are more likely to aggregate global and local information as well as information between channels than other combinations. Our method achieved an average Jaccard Index of 0.7692 on the ISIC2017 dataset. At the same time, we also compared with some advanced methods of image segmentation, the experimental results show our proposed method has a competitive performance.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

References

  1. Badrinarayanan V, Kendall A, Cipolla R (2017) SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell 39:2481–2495. https://doi.org/10.1109/TPAMI.2016.2644615

    Article  Google Scholar 

  2. Bi L, Kim J, Ahn E, Kumar A, Fulham M, Feng D (2017) Dermoscopic image segmentation via multistage fully convolutional networks. IEEE Trans Biomed Eng 64:2065–2074. https://doi.org/10.1109/TBME.2017.2712771

    Article  Google Scholar 

  3. Chaki J, Dey N, Rajinikanth V, Ashour AS, Shi F (2021) Recognition of skin diseases using curvelet transforms and law’s texture energy measures. Soft computing applications. Springer International Publishing, Cham, pp 51–61

    Google Scholar 

  4. Chen L-C, Papandreou G, Schroff F, Adam HJA (2017) Rethinking atrous convolution for semantic image segmentation abs/1706.05587

  5. Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H (2018a) Encoder-decoder with atrous separable convolution for semantic image segmentation. Computer vision—ECCV 2018. Springer International Publishing, Cham, pp 833–851

    Google Scholar 

  6. Chen L, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2018b) DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Mach Intell 40:834–848. https://doi.org/10.1109/TPAMI.2017.2699184

    Article  Google Scholar 

  7. Codella NCF et al. (2018) Skin lesion analysis toward melanoma detection: a challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), 4–7 April 2018, pp 168–172. https://doi.org/10.1109/ISBI.2018.8363547

  8. Dey N, Rajinikanth V, Ashour AS, Tavares JMRS (2018) Social group optimization supported segmentation and evaluation of skin melanoma images. Symmetry. https://doi.org/10.3390/sym10020051

    Article  MATH  Google Scholar 

  9. Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2019) Dual attention network for scene segmentation. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), 15–20 June 2019, pp 3141–3149. https://doi.org/10.1109/CVPR.2019.00326

  10. Garcia-Arroyo JL, Garcia-Zapirain B (2019) Segmentation of skin lesions in dermoscopy images using fuzzy classification of pixels and histogram thresholding. Comput Methods Programs Biomed 168:11–19. https://doi.org/10.1016/j.cmpb.2018.11.001

    Article  Google Scholar 

  11. Gu Z et al (2019) CE-Net: context encoder network for 2D medical image segmentation. IEEE Trans Med Imaging 38:2281–2292. https://doi.org/10.1109/TMI.2019.2903562

    Article  Google Scholar 

  12. He K, Zhang X, Ren S, Sun J (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37:1904–1916. https://doi.org/10.1109/TPAMI.2015.2389824

    Article  Google Scholar 

  13. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), 27–30 June 2016, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  14. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, 18–23 June 2018, pp 7132–7141. https://doi.org/10.1109/CVPR.2018.00745

  15. Hu J, Shen L, Albanie S, Sun G, Wu E (2020) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 42:2011–2023. https://doi.org/10.1109/TPAMI.2019.2913372

    Article  Google Scholar 

  16. Jha D, Riegler MA, Johansen D, Halvorsen P, Johansen HD (2020) DoubleU-Net: a deep convolutional neural network for medical image segmentation. In: 2020 IEEE 33rd International symposium on computer-based medical systems (CBMS), 28–30 July 2020, pp 558–564. https://doi.org/10.1109/CBMS49503.2020.00111

  17. Kaul C, Manandhar S, Pears N (2019) Focusnet: an attention-based fully convolutional network for medical image segmentation. In: 2019 IEEE 16th international symposium on biomedical imaging (ISBI 2019), 8–11 April 2019, pp 455–458. https://doi.org/10.1109/ISBI.2019.8759477

  18. Li Y, Shen L (2018) Skin lesion analysis towards melanoma detection using deep learning network. Sensors. https://doi.org/10.3390/s18020556

    Article  Google Scholar 

  19. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), 7–12 June 2015, pp 3431–3440. https://doi.org/10.1109/CVPR.2015.7298965

  20. Md Zahangir A, Chris Y, Mahmudul H, Tarek MT, Vijayan KA (2019) Recurrent residual U-Net for medical image segmentation. J Med Imaging 6:1–16. https://doi.org/10.1117/1.JMI.6.1.014006

    Article  Google Scholar 

  21. Peng C, Zhang X, Yu G, Luo G, Sun J (2017) Large Kernel matters—improve semantic segmentation by global convolutional network. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), 21–26 July 2017, pp 1743–1751. https://doi.org/10.1109/CVPR.2017.189

  22. Rahman M, Alpaslan N, Bhattacharya P (2016) Developing a retrieval based diagnostic aid for automated melanoma recognition of dermoscopic images. In: 2016 IEEE applied imagery pattern recognition workshop (AIPR), 18–20 Oct. 2016, pp 1–7. https://doi.org/10.1109/AIPR.2016.8010594

  23. Rajinikanth V, Satapathy SC, Dey N, Fernandes SL, Manic KS (2019) Skin melanoma assessment using Kapur’s entropy and level set—a study with bat algorithm. Smart intelligent computing and applications. Springer Singapore, Singapore, pp 193–202

    Google Scholar 

  24. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. Medical image computing and computer-assisted intervention—MICCAI 2015. Springer International Publishing, Cham, pp 234–241

    Google Scholar 

  25. Roy AG, Navab N, Wachinger C (2018) Concurrent spatial and channel ‘squeeze & excitation’ in fully convolutional networks. Medical image computing and computer assisted intervention—MICCAI 2018. Springer International Publishing, Cham, pp 421–429

    Google Scholar 

  26. Silveira M et al (2009) Comparison of segmentation methods for melanoma diagnosis in dermoscopy images. IEEE J Sel Top Signal Process 3:35–45. https://doi.org/10.1109/JSTSP.2008.2011119

    Article  Google Scholar 

  27. Society AC (2020) Key statistics for melanoma skin cancer. https://www.cancer.org/cancer/melanoma-skin-cancer/about/keystatistics.html. Accessed 4 Jul 2020

  28. Sun K, Xiao B, Liu D, Wang J (2019) Deep high-resolution representation learning for human pose estimation. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), 15–20 June 2019, pp 5686–5696. https://doi.org/10.1109/CVPR.2019.00584

  29. Thao LT, Quang NH (2017) Automatic skin lesion analysis towards melanoma detection. In: 2017 21st Asia Pacific symposium on intelligent and evolutionary systems (IES), 15–17 Nov. 2017, pp 106–111. https://doi.org/10.1109/IESYS.2017.8233570

  30. Vaswani A et al. (2017) Attention is all you need. In: Paper presented at the proceedings of the 31st international conference on neural information processing systems, Long Beach, California, USA

  31. Wang X, Girshick R, Gupta A, He K (2018) Non-local neural networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, 18–23 June 2018, pp 7794–7803. https://doi.org/10.1109/CVPR.2018.00813

  32. Wang Q, Wu B, Zhu P, Li P, Zuo W, Hu Q (2020) ECA-Net: efficient channel attention for deep convolutional neural networks. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), 13–19 June 2020, pp 11531–11539. https://doi.org/10.1109/CVPR42600.2020.01155

  33. Wen HJA (2017) II-FCN for skin lesion analysis towards melanoma detection. abs/1702.08699

  34. Wong A, Scharcanski J, Fieguth P (2011) Automatic skin lesion segmentation via iterative stochastic region merging. IEEE Trans Inf Technol Biomed 15:929–936. https://doi.org/10.1109/TITB.2011.2157829

    Article  Google Scholar 

  35. Yang M, Yu K, Zhang C, Li Z, Yang K (2018) DenseASPP for semantic segmentation in street scenes. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, 18–23 June 2018. pp 3684–3692. https://doi.org/10.1109/CVPR.2018.00388

  36. Yu L, Chen H, Dou Q, Qin J, Heng P (2017b) Automated melanoma recognition in dermoscopy images via very deep residual networks. IEEE Trans Med Imaging 36:994–1004. https://doi.org/10.1109/TMI.2016.2642839

    Article  Google Scholar 

  37. Yu F, Koltun V, Funkhouser T (2017a) Dilated residual networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), 21–26 July 2017, pp 636–644. https://doi.org/10.1109/CVPR.2017.75

  38. Yuan Y, Lo YC (2019) Improving dermoscopic image segmentation with enhanced convolutional-deconvolutional networks. IEEE J Biomed Health Inform 23:519–526. https://doi.org/10.1109/JBHI.2017.2787487

    Article  Google Scholar 

  39. Yuan Y, Chao M, Lo Y (2017) Automatic skin lesion segmentation using deep fully convolutional networks with Jaccard distance. IEEE Trans Med Imaging 36:1876–1886. https://doi.org/10.1109/TMI.2017.2695227

    Article  Google Scholar 

  40. Zhang H, Goodfellow I, Metaxas D, Odena A (2019) Self-attention generative adversarial networks. In: Paper presented at the proceedings of the 36th international conference on machine learning, proceedings of machine learning research

  41. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), 21–26 July 2017, pp 6230–6239. https://doi.org/10.1109/CVPR.2017.660

  42. Zhou H, Li X, Schaefer G, Celebi ME, Miller P (2013) Mean shift based gradient vector flow for image segmentation. Comput Vis Image Underst 117:1004–1016. https://doi.org/10.1016/j.cviu.2012.11.015

    Article  Google Scholar 

  43. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J (2020) UNet++: redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imaging 39:1856–1867. https://doi.org/10.1109/TMI.2019.2959609

    Article  Google Scholar 

Download references

Acknowledgements

This research is partially supported by Science and Technology Department of Xinjiang Uyghur Autonomous Region Fund Project (2020E0234), and Department of Education, Xinjiang Uygur Autonomous Region (CN) Postgraduate Research and Innovation Project (XJ2020G072). We would also like to thank our tutor for the careful guidance and all the participants for their insightful comments.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Long Yu.

Ethics declarations

Conflict of interest

The authors declare no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ren, Y., Yu, L., Tian, S. et al. Serial attention network for skin lesion segmentation. J Ambient Intell Human Comput (2021). https://doi.org/10.1007/s12652-021-02933-3

Download citation

Keywords

  • Spatial attention
  • Channel attention
  • Skin lesion segmentation
  • Deep learning