Abstract
This paper presents a work consisting in using deep convolutional neural networks (CNNs) to facilitate the curation of brand-related social media images. The final goal is to facilitate searching and discovering user-generated content (UGC) with potential value for digital marketing tasks. The images are captured in real time and automatically annotated with multiple CNNs. Some of the CNNs perform generic object recognition tasks while others perform what we call visual brand identity recognition. When appropriate, we also apply object detection, usually to discover images containing logos. We report experiments with 5 real brands in which more than 1 million real images were analyzed. In order to speed-up the training of custom CNNs we applied a transfer learning strategy. We examine the impact of different configurations and derive conclusions aiming to pave the way towards systematic and optimized methodologies for automatic UGC curation.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bakhshi S, Shamma D A, Kennedy L, Gilbert E (2015) Why we filter our photos and how it impacts engagement. In: Proceedings of the ninth international conference on web and social media, ICWSM 2015. University of Oxford, Oxford, pp 12–21
Chute Enterprise ugc. http://www.getchute.com/. Accessed 6 June (2017)
Clarke T, Costall A (2008) The emotional connotations of color: a qualitative investigation. Color Res Appl 33(5):406–410
Clark M, Black H G, Judson K (2017) Brand community integration and satisfaction with social media sites: a comparative study. J Res Interact Market 11 (1):39–55
Curalate https://www.curalate.com/. Accessed 6 June (2017)
Denton E, Weston J, Paluri M, Bourdev L, Fergus R (2015) User conditional hashtag prediction for images. In: Proceedings of the 21th ACM SIGKDD International conference on knowledge discovery and data mining, KDD ’15. ACM, pp 1731–1740
Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T (2014) Decaf: a deep convolutional activation feature for generic visual recognition. In: Proceedings of the 31th international conference on machine learning, ICML 2014. Beijing, China, 21-26 June 2014, pp 647–655
Gao Y, Wang F, Luan H, Chua T (2014) Brand data gathering from live social media streams. In: Proceedings of the international conference on multimedia retrieval, ICMR 2014. Glasgow, United Kingdom - April 01 - 04, 2014, p 169
Gao Y, Zhao S, Yang Y, Chua T (2015) Multimedia social event detection in microblog. In: MultiMedia Modeling - 21st International conference, MMM 2015. Sydney, NSW, Australia, January 5-7, 2015, Proceedings, Part I, pp 269–281
Gao Y, Zhen Y, Li H, Chua T (2016) Filtering of brand-related microblogs using social-smooth multiview embedding. IEEE Trans Multimed 18(10):2115–2126
Nguyen DT, Alam F, Ofli F, Imran M (2017) Automatic image filtering on social networks using deep learning and perceptual hashing during crises. CoRR arXiv:1704.02602
Olapic Earned content platform. http://www.olapic.com/. Accessed 6 June (2017)
Park M, Li H, Kim J (2016) HARRISON: a benchmark on hashtag recommendation for real-world images in social networks. CoRR arXiv:1605.05054
Redmon J, Divvala S K, Girshick R B, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the 2016 IEEE conference on computer vision and pattern recognition, CVPR 2016. Las Vegas, NV, USA, June 27-30, 2016, pp 779–788
Sermanet P, Eigen D, Zhang X, Mathieu M, Fergus R, Lecun Y (2014) Overfeat: integrated recognition, localization and detection using convolutional networks. In: Proceedings of the international conference on learning representations (ICLR2014), CBLS, April 2014
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the 2016 IEEE conference on computer vision and pattern recognition, CVPR 2016. Las Vegas, NV, USA, June 27-30, 2016, pp 2818–2826
Tous R, Torres J, Ayguadé E (2015) Multimedia big data computing for in-depth event analysis. In: Proceedings of the 2015 IEEE International conference on multimedia big data (BigMM), April 20-22, 2015, Beijing, China. IEEE, pp 144–147
Tous R, Wu̇st O, Gomez M, Poveda J, Elena M, Torres J, Makni M, Ayguadė E (2016) User-generated content curation with deep convolutional neural networks. In: Proceedings of the 2016 IEEE international conference on big data, BigData 2016. Washington DC, USA, December 5-8, 2016, pp 2535– 2540
Wang F, Qi S, Gao G, Zhao S, Wang X (2016) Logo information recognition in large-scale social media data. Multimed Syst 22(1):63–73
Zhao S, Yao H, Gao Y, Ji R, Xie W, Jiang X, Chua T (2016) Predicting personalized emotion perceptions of social images. In: Proceedings of the 2016 ACM conference on multimedia conference, MM 2016. Amsterdam, The Netherlands, October 15-19, 2016, pp 1385–1394
Zhao S, Yao H, Zhao S, Jiang X, Jiang X (2016) Multi-modal microblog classification via multi-task learning. Multimedia Tools Appl 75(15):8921–8938
Acknowledgements
This work is partially supported by the Spanish Ministry of Economy and Competitivity under contract TIN2015-65316-P and by the SGR programme (2014-SGR-1051) of the Catalan Government.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Tous, R., Gomez, M., Poveda, J. et al. Automated curation of brand-related social media images with deep learning. Multimed Tools Appl 77, 27123–27142 (2018). https://doi.org/10.1007/s11042-018-5910-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-018-5910-z