Advertisement

FashionSearchNet: Fashion Search with Attribute Manipulation

  • Kenan E. AkEmail author
  • Ashraf A. Kassim
  • Joo Hwee Lim
  • Jo Yew Tham
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11131)

Abstract

The focus of this paper is on retrieval of fashion images after manipulating attributes of the query images. This task is particularly useful in search scenarios where the user is interested in small variations of an image, i.e., replacing the mandarin collar with a buttondown. Keeping the desired attributes of the query image while manipulating its other attributes is a challenging problem which is accomplished by our proposed network called FashionSearchNet. FashionSearchNet is able to learn attribute specific representations by leveraging on weakly-supervised localization. The localization module is used to ignore the unrelated features of attributes in the feature map, thus improve the similarity learning. Experiments conducted on two recent fashion datasets show that FashionSearchNet outperforms the other state-of-the-art fashion search techniques.

Keywords

CNNs Fashion retrieval Similarity learning Attribute localization 

References

  1. 1.
    Dong, Q., Gong, S., Zhu, X.: Multi-task curriculum transfer deep learning of clothing attributes. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 520–529 (2017)Google Scholar
  2. 2.
    Liu, Z., Luo, P., Qiu, S., Wang, X., Tang, X.: Deepfashion: powering robust clothes recognition and retrieval with rich annotations. In: IEEE Conference Computer Vision Pattern Recognition (CVPR), pp. 1096–1104 (2016)Google Scholar
  3. 3.
    Simo-Serra, E., Ishikawa, H.: Fashion style in 128 floats: joint ranking and classification using weak data for feature extraction. In: IEEE Conference Computer Vision Pattern Recognition (CVPR), pp. 298–307 (2016)Google Scholar
  4. 4.
    Corbiere, C., Ben-Younes, H., Ramé, A., Ollion, C.: Leveraging weakly annotated data for fashion image retrieval and label prediction. arXiv preprint arXiv:1709.09426 (2017)
  5. 5.
    Hadi Kiapour, M., Han, X., Lazebnik, S., Berg, A.C., Berg, T.L.: Where to buy it: matching street clothing photos in online shops. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 3343–3351 (2015)Google Scholar
  6. 6.
    Han, X., et al.: Automatic spatially-aware fashion concept discovery. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1463–1471 (2017)Google Scholar
  7. 7.
    Vittayakorn, S., Umeda, T., Murasaki, K., Sudo, K., Okatani, T., Yamaguchi, K.: Automatic attribute discovery with neural activations. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 252–268. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_16CrossRefGoogle Scholar
  8. 8.
    Liang, X., et al.: Human parsing with contextualized convolutional neural network. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1386–1394 (2015)Google Scholar
  9. 9.
    Simo-Serra, E., Fidler, S., Moreno-Noguer, F., Urtasun, R.: A high performance CRF model for clothes parsing. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9005, pp. 64–81. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-16811-1_5CrossRefGoogle Scholar
  10. 10.
    Yamaguchi, K., Hadi Kiapour, M., Berg, T.L.: Paper doll parsing: retrieving similar styles to parse clothing items. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 3519–3526 (2013)Google Scholar
  11. 11.
    Al-Halah, Z., Stiefelhagen, R., Grauman, K.: Fashion forward: Forecasting visual style in fashion. arXiv preprint arXiv:1705.06394 (2017)
  12. 12.
    Han, X., Wu, Z., Jiang, Y.G., Davis, L.S.: Learning fashion compatibility with bidirectional lstms. arXiv preprint arXiv:1707.05691 (2017)
  13. 13.
    Simo-Serra, E., Fidler, S., Moreno-Noguer, F., Urtasun, R.: Neuroaesthetics in fashion: modeling the perception of fashionability. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 869–877 (2015)Google Scholar
  14. 14.
    Zhu, S., Fidler, S., Urtasun, R., Lin, D., Loy, C.C.: Be your own prada: fashion synthesis with structural coherence. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1689–1697. IEEE (2017)Google Scholar
  15. 15.
    Han, X., Wu, Z., Wu, Z., Yu, R., Davis, L.S.: Viton: an image-based virtual try-on network. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018Google Scholar
  16. 16.
    Zhao, B., Feng, J., Wu, X., Yan, S.: Memory-augmented attribute manipulation networks for interactive fashion search. In: IEEE Conference Computer Vision Pattern Recognition (CVPR) (2017)Google Scholar
  17. 17.
    Ak, K.E., Kassim, A.A., Lim, J.H., Tham, J.Y.: Learning attribute representations with localization for flexible fashion search. In: IEEE Conference Computer Vision Pattern Recognition, CVPR, pp. 7708–7717 (2018)Google Scholar
  18. 18.
    Chen, Q., Huang, J., Feris, R., Brown, L.M., Dong, J., Yan, S.: Deep domain adaptation for describing people based on fine-grained clothing attributes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 7, pp. 5315–5324 (2015)Google Scholar
  19. 19.
    Yamaguchi, K., Okatani, T., Sudo, K., Murasaki, K., Taniguchi, Y.: Mix and match: joint model for clothing and attribute recognition. In: BMVC, p. 51–1 (2015)Google Scholar
  20. 20.
    Abdulnabi, A.H., Wang, G., Lu, J., Jia, K.: Multi-task CNN model for attribute prediction. IEEE Trans. Multimedia 17(11), 1949–1959 (2015)CrossRefGoogle Scholar
  21. 21.
    Song, Y., Li, Y., Wu, B., Chen, C.Y., Zhang, X., Adam, H.: Learning unified embedding for apparel recognition. arXiv preprint arXiv:1707.05929 (2017)
  22. 22.
    Huang, J., Feris, R.S., Chen, Q., Yan, S.: Cross-domain image retrieval with a dual attribute-aware ranking network. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1062–1070 (2015)Google Scholar
  23. 23.
    Hsiao, W.L., Grauman, K.: Creating capsule wardrobes from fashion images. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018Google Scholar
  24. 24.
    Tangseng, P., Yamaguchi, K., Okatani, T.: Recommending outfits from personal closet. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2275–2279 (2017)Google Scholar
  25. 25.
    Singh, K.K., Lee, Y.J.: End-to-end localization and ranking for relative attributes. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 753–769. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46466-4_45CrossRefGoogle Scholar
  26. 26.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp. 1097–1105 (2012)Google Scholar
  27. 27.
    Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)Google Scholar
  28. 28.
    Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)Google Scholar
  29. 29.
    Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: Feragen, A., Pelillo, M., Loog, M. (eds.) SIMBAD 2015. LNCS, vol. 9370, pp. 84–92. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24261-3_7CrossRefGoogle Scholar
  30. 30.
    Ak, K.E., Lim, J.H., Tham, J.Y., Kassim, A.A.: Efficient multi-attribute similarity learning towards attribute-based fashion search. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1671–1679. IEEE (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Kenan E. Ak
    • 1
    • 2
    Email author
  • Ashraf A. Kassim
    • 1
  • Joo Hwee Lim
    • 2
  • Jo Yew Tham
    • 3
  1. 1.National University of SingaporeSingaporeSingapore
  2. 2.Institute for Infocomm Research, A*STARSingaporeSingapore
  3. 3.ESP xMedia Pte. Ltd.SingaporeSingapore

Personalised recommendations