Advertisement

EL-GAN: Embedding Loss Driven Generative Adversarial Networks for Lane Detection

  • Mohsen GhafoorianEmail author
  • Cedric Nugteren
  • Nóra Baka
  • Olaf Booij
  • Michael Hofmann
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11129)

Abstract

Convolutional neural networks have been successfully applied to semantic segmentation problems. However, there are many problems that are inherently not pixel-wise classification problems but are nevertheless frequently formulated as semantic segmentation. This ill-posed formulation consequently necessitates hand-crafted scenario-specific and computationally expensive post-processing methods to convert the per pixel probability maps to final desired outputs. Generative adversarial networks (GANs) can be used to make the semantic segmentation network output to be more realistic or better structure-preserving, decreasing the dependency on potentially complex post-processing.

In this work, we propose EL-GAN: a GAN framework to mitigate the discussed problem using an embedding loss. With EL-GAN, we discriminate based on learned embeddings of both the labels and the prediction at the same time. This results in much more stable training due to having better discriminative information, benefiting from seeing both ‘fake’ and ‘real’ predictions at the same time. This substantially stabilizes the adversarial training process. We use the TuSimple lane marking challenge to demonstrate that with our proposed framework it is viable to overcome the inherent anomalies of posing it as a semantic segmentation problem. Not only is the output considerably more similar to the labels when compared to conventional methods, the subsequent post-processing is also simpler and crosses the competitive 96% accuracy threshold.

Notes

Acknowledgments

The authors would like to thank Nicolau Leal Werneck, Stefano Secondo, Jihong Ju, Yu Wang, Sindi Shkodrani and Bram Beernink for their contributions and valuable feedback.

Supplementary material

478770_1_En_15_MOESM1_ESM.pdf (91 kb)
Supplementary material 1 (pdf 91 KB)
478770_1_En_15_MOESM2_ESM.mp4 (22.1 mb)
Supplementary material 2 (mp4 22608 KB)

References

  1. 1.
    Bar Hillel, A., Lerner, R., Levi, D., Raz, G.: Recent progress in road and lane detection: a survey. Mach. Vis. Appl. 25(3), 727–745 (2014).  https://doi.org/10.1007/s00138-011-0404-2CrossRefGoogle Scholar
  2. 2.
    BenTaieb, A., Hamarneh, G.: Topology aware fully convolutional networks for histology gland segmentation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 460–468. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46723-8_53CrossRefGoogle Scholar
  3. 3.
    Dai, W., et al.: Scan: structure correcting adversarial network for chest x-rays organ segmentation. arXiv preprint arXiv:1703.08770 (2017)
  4. 4.
    Dosovitskiy, A., Brox, T.: Generating images with perceptual similarity metrics based on deep networks. In: NIPS: Advances in Neural Information Processing Systems, pp. 658–666 (2016)Google Scholar
  5. 5.
    Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 2672–2680. Curran Associates, Inc. (2014). http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf
  6. 6.
    Huang, G., Liu, Z., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR: Computer Vision and Pattern Recognition (2017)Google Scholar
  7. 7.
    Hung, W.C., Tsai, Y.H., Liou, Y.T., Lin, Y.Y., Yang, M.H.: Adversarial learning for semi-supervised semantic segmentation. arXiv e-prints, February 2018Google Scholar
  8. 8.
    Huo, Y., et al.: Splenomegaly segmentation using global convolutional kernels and conditional generative adversarial networks. In: Proceedings of SPIE, vol. 10574, pp. 10574–10574-7 (2018).  https://doi.org/10.1117/12.2293406
  9. 9.
    Hwang, J.J., Ke, T.W., Shi, J., Yu, S.X.: Adversarial structure matching loss for image segmentation. arXiv preprint arXiv:1805.07457 (2018)
  10. 10.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR: Computer Vision and Pattern Recognition (2017)Google Scholar
  11. 11.
    Jegou, S., Drozdzal, M., Vazquez, D., Romero, A., Bengio, Y.: The one hundred layers tiramisu: fully convolutional DenseNets for semantic segmentation. In: CVPRW: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1175–1183, July 2017.  https://doi.org/10.1109/CVPRW.2017.156
  12. 12.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_43CrossRefGoogle Scholar
  13. 13.
    Kohl, S., et al.: Adversarial networks for the detection of aggressive prostate cancer. arXiv preprint arXiv:1702.08014 (2017)
  14. 14.
    Krähenbühl, P., Koltun, V.: Efficient inference in fully connected CRFs with Gaussian edge potentials. In: Advances in Neural Information Processing Systems, pp. 109–117 (2011)Google Scholar
  15. 15.
    Lee, S., et al.: VPGNet: vanishing point guided network for lane and road marking detection and recognition. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1965–1973. IEEE (2017)Google Scholar
  16. 16.
    Li, C., Wand, M.: Precomputed real-time texture synthesis with markovian generative adversarial networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 702–716. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46487-9_43CrossRefGoogle Scholar
  17. 17.
    Li, Z., Wang, Y., Yu, J.: Brain tumor segmentation using an adversarial network. In: Crimi, A., Bakas, S., Kuijf, H., Menze, B., Reyes, M. (eds.) BrainLes 2017. LNCS, vol. 10670, pp. 123–132. Springer, Cham (2018).  https://doi.org/10.1007/978-3-319-75238-9_11CrossRefGoogle Scholar
  18. 18.
    Luc, P., Couprie, C., Chintala, S., Verbeek, J.: Semantic segmentation using adversarial networks. In: NIPS Workshop on Adversarial Training, Barcelona, Spain, December 2016. https://hal.inria.fr/hal-01398049
  19. 19.
    Moeskops, P., Veta, M., Lafarge, M.W., Eppenhof, K.A.J., Pluim, J.P.W.: Adversarial training and dilated convolutions for brain MRI segmentation. In: Cardoso, M.J., et al. (eds.) DLMIA/ML-CDS -2017. LNCS, vol. 10553, pp. 56–64. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-67558-9_7CrossRefGoogle Scholar
  20. 20.
    Neven, D., De Brabandere, B., Georgoulis, S., Proesmans, M., Van Gool, L.: Towards end-to-end lane detection: an instance segmentation approach. arXiv e-prints, February 2018Google Scholar
  21. 21.
    Nguyen, V., Vicente, T.F.Y., Zhao, M., Hoai, M., Samaras, D.: Shadow detection with conditional generative adversarial networks. In: ICCV: IEEE International Conference on Computer Vision, pp. 4520–4528. IEEE (2017)Google Scholar
  22. 22.
    Oktay, O., et al.: Anatomically constrained neural networks (ACNN): application to cardiac image enhancement and segmentation. IEEE Trans. Med. Imaging 37(2), 384–395 (2017)CrossRefGoogle Scholar
  23. 23.
    Pan, X., Shi, J., Luo, P., Wang, X., Tang, X.: Spatial as deep: spatial CNN for traffic scene understanding. In: AAAI Conference on Artificial Intelligence, February 2018Google Scholar
  24. 24.
    Sadanandan, S.K., Karlsson, J., Whlby, C.: Spheroid segmentation using multiscale deep adversarial networks. In: ICCVW: IEEE International Conference on Computer Vision Workshops, pp. 36–41, October 2017.  https://doi.org/10.1109/ICCVW.2017.11
  25. 25.
    Sajjadi, M.S., Scholkopf, B., Hirsch, M.: EnhanceNet: single image super-resolution through automated texture synthesis. In: CVPR: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4491–4500 (2017)Google Scholar
  26. 26.
    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, pp. 2234–2242 (2016)Google Scholar
  27. 27.
    Schwing, A.G., Urtasun, R.: Fully connected deep structured networks. arXiv e-prints, March 2015Google Scholar
  28. 28.
    Xue, Y., Xu, T., Zhang, H., Long, R., Huang, X.: SegAN: adversarial network with multi-scale \(L\_1\) loss for medical image segmentation. arXiv e-prints, June 2017Google Scholar
  29. 29.
    Yang, D., et al.: Automatic liver segmentation using an adversarial image-to-image network. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 507–515. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-66179-7_58CrossRefGoogle Scholar
  30. 30.
    Zhang, Y., Yang, L., Chen, J., Fredericksen, M., Hughes, D.P., Chen, D.Z.: Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 408–416. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-66179-7_47CrossRefGoogle Scholar
  31. 31.
    Zheng, S., et al.: Conditional random fields as recurrent neural networks. In: ICCV: International Conference on Computer Vision, pp. 1529–1537 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Mohsen Ghafoorian
    • 1
    Email author
  • Cedric Nugteren
    • 1
  • Nóra Baka
    • 1
  • Olaf Booij
    • 1
  • Michael Hofmann
    • 1
  1. 1.TomTomAmsterdamThe Netherlands

Personalised recommendations