Advertisement

Human-Centric Visual Relation Segmentation Using Mask R-CNN and VTransE

  • Fan Yu
  • Xin Tan
  • Tongwei RenEmail author
  • Gangshan Wu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11130)

Abstract

In this paper, we propose a novel human-centric visual relation segmentation method based on Mask R-CNN model and VTransE model. We first retain the Mask R-CNN model, and segment both human and object instances. Because Mask R-CNN may omit some human instances in instance segmentation, we further detect the omitted faces and extend them to localize the corresponding human instances. Finally, we retrain the last layer of VTransE model, and detect the visual relations between each pair of human instance and human/object instance. The experimental results show that our method obtains 0.4799, 0.4069, and 0.2681 on the criteria of R@100 with the m-IoU of 0.25, 0.50 and 0.75, respectively, which outperforms other methods in Person in Context Challenge.

Keywords

Human-centric Visual relation segmentation Mask R-CNN VTransE 

Notes

Acknowledgements

This work is supported by National Science Foundation of China (61202320), the Fundamental Research Funds for the Central Universities (021714380011), and Collaborative Innovation Center of Novel Software Technology and Industrialization.

Supplementary material

478816_1_En_44_MOESM1_ESM.zip (91.4 mb)
Supplementary material 1 (zip 93609 KB)

References

  1. 1.
    He, K., Gkioxari, G., Dollr, P., Girshick, R.: Mask R-CNN. In: IEEE International Conference on Computer Vision. IEEE (2017)Google Scholar
  2. 2.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Conference and Workshop on Neural Information Processing Systems. MIT Press (2015)Google Scholar
  3. 3.
    Zhang, H., Kyaw, Z., Chang, S.-F., Chua, T.-S.: Visual translation embedding network for visual relation detection. In: IEEE Conference on Computer Vision and Pattern Recognition. IEEE (2017)Google Scholar
  4. 4.
    Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on Computer Vision and Pattern Recognition. IEEE (2014)Google Scholar
  5. 5.
    Girshick, R.: Fast R-CNN. In: IEEE International Conference on Computer Vision. IEEE (2015)Google Scholar
  6. 6.
    Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: Conference and Workshop on Neural Information Processing Systems. MIT Press (2013)Google Scholar
  7. 7.
    Lu, C., Krishna, R., Bernstein, M., Fei-Fei, L.: Visual relationship detection with language priors. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 852–869. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46448-0_51CrossRefGoogle Scholar
  8. 8.
    Liang, X., Lee, L., Xing, E.P.: Deep variation-structured reinforcement learning for visual relationship and attribute detection. In: IEEE Conference on Computer Vision and Pattern Recognition. IEEE (2017)Google Scholar
  9. 9.
    Shang, X., Ren, T., Guo, J., Zhang, H., Chua, T.-S.: Video visual relation detection. In: ACM International Conference on Multimedia. ACM (2017)Google Scholar
  10. 10.
    Li, X., Song, X., Herranz, L., Zhu, Y., Shuqiang, J.: Image captioning with both object and scene information. In: ACM International Conference on Multimedia. ACM (2016)Google Scholar
  11. 11.
    Antol, S., et al.: VQA: visual question answering. In: IEEE International Conference on Computer Vision. IEEE (2015)Google Scholar
  12. 12.
    Guo, H., Wang, J., Xu, M., Zha, Z.-J., Lu, H.: Learning multi-view deep features for small object retrieval in surveillance scenarios. In: ACM International Conference on Multimedia. ACM (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.State Key Laboratory for Novel Software TechnologyNanjing UniversityNanjingChina

Personalised recommendations