Crowdsourced object-labeling based on a game-based mobile application

  • Wen-Yen Tseng
  • Kai-Hsiang ChenEmail author
  • Jen-Wei Huang


Unparalleled growth in the sharing of media via networks has prompted a great deal of research into issues pertaining to image retrieval. The training and verification of image retrieval systems requires a large number of labelled images with ground truth; however, most researchers employ public datasets for their experiments, the results are restricted by the size and content of the dataset. In this study, we developed a system based on a mobile phone App for the collection of information pertaining to the location of objects in images. The proposed system is simple and easy to use. Experiments demonstrate the excellent performance of the proposed system with regard to accuracy and response time. This study demonstrates the feasibility of collecting image information using mobile phones.


Interactive system Crowd sourced Object labelling Data collection 



  1. 1.
    Bannour H, Hudelot C (2012) Hierarchical image annotation using semantic hierarchies. In: Proceedings of the 21st ACM international conference on information and knowledge management. ACM, pp 2431–2434Google Scholar
  2. 2.
    Captcha forms – good or evil? (2012).
  3. 3.
    Chellapilla K, Larson K, Simard P, Czerwinski M (2005) Designing human friendly human interaction proofs (hips). In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 711–720Google Scholar
  4. 4.
    Chen KH, Wu CR, Yang Y, Huang JW, Ho TY (2014) Efficient building identification using structural and spatial information on mobile devices. In: 2014 IEEE international conference on multimedia and expo workshops (ICMEW). IEEE, pp 1–5Google Scholar
  5. 5.
    Customizing the look and feel of recaptcha (2016).
  6. 6.
    Datta R, Joshi D, Li J, Wang J (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv (CSUR) 40(2):5CrossRefGoogle Scholar
  7. 7.
    Eom W, Lee S, De Neve W, Ro YM (2011) Improving image tag recommendation using favorite image context. In: 2011 18th IEEE international conference on image processing. IEEE, pp 2445–2448Google Scholar
  8. 8.
    Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results.
  9. 9.
    Gao Y, Peng J, Luo H, Keim DA, Fan J (2009) An interactive approach for filtering out junk images from keyword-based google search results. IEEE Trans Circ Syst Video Technol 19(12):1851–1865CrossRefGoogle Scholar
  10. 10.
    Google now using recaptcha to decode street view addresses (2012).
  11. 11.
    Huang L, Yang Y, Deng Y, Yu Y (2015) Densebox: unifying landmark localization with end to end object detection. arXiv:1509.04874
  12. 12.
    Jing P, Su Y, Nie L, Gu H, Liu J, Wang M (2018) A framework of joint low-rank and sparse regression for image memorability prediction. IEEE Transactions on Circuits and Systems for Video Technology, pp 1–1.
  13. 13.
    Joachims T (2002) Optimizing search engines using clickthrough data. In: Proceedings of the eighth ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 133–142Google Scholar
  14. 14.
    Lerthirunwong N, Shimizu I (2012) Interactive metric learning system for similar image search using linear discriminant analysis. In: The 1st IEEE global conference on consumer electronics 2012. IEEE, pp 206–209Google Scholar
  15. 15.
    Li B, Xiong W, Hu W (2011) Web horror image recognition based on context-aware multi-instance learning. In: 2011 IEEE 11th international conference on data mining. IEEE, pp 1158–1163Google Scholar
  16. 16.
    Liu Y, Mei T (2011) Optimizing visual search reranking via pairwise learning. IEEE Trans Multimed 13(2):280–291CrossRefGoogle Scholar
  17. 17.
    Liu Y, Mei T, Hua XS (2009) Crowdreranking: exploring multiple search engines for visual search reranking. In: Proceedings of the 32nd international ACM SIGIR conference on research and development in information retrieval. ACM, pp 500–507Google Scholar
  18. 18.
    Liu D, Hua XS, Wang M, Zhang HJ (2010) Image retagging. In: Proceedings of the 18th ACM international conference on multimedia. ACM, pp 491–500Google Scholar
  19. 19.
    Liu Y, Wu F, Zhang Y, Shao J, Zhuang Y (2011) Tag clustering and refinement on semantic unity graph. In: 2011 IEEE 11th international conference on data mining. IEEE, pp 417–426Google Scholar
  20. 20.
    Liu X, Qian X, Lu D, Hou X, Wang L (2014) Personalized tag recommendation for flickr users. In: 2014 IEEE International conference on multimedia and expo (ICME). IEEE, pp 1–6Google Scholar
  21. 21.
    Philbin J, Chum O, Isard M, Sivic J, Zisserman A (2007) Object retrieval with large vocabularies and fast spatial matching. In: 2007 IEEE conference on computer vision and pattern recognition. IEEE, pp 1–8Google Scholar
  22. 22.
    Rae A, Sigurbjörnsson B, van Zwol R (2010) Improving tag recommendation using social networks. In: Adaptivity, personalization and fusion of heterogeneous information. Le Centre De Hautes Etudes Internationales D’Informatique Documentaire, pp 92–99Google Scholar
  23. 23.
    recaptcha – i’m not a robot (2017).
  24. 24.
    Russell BC, Torralba A, Murphy KP, Freeman WT (2008) Labelme: a database and web-based tool for image annotation. Int J Comput Vis 77(1-3):157–173CrossRefGoogle Scholar
  25. 25.
    Su JH, Chou CL, Lin CY, Tseng VS (2011) Effective semantic annotation by image-to-concept distribution model. IEEE Trans Multimed 13(3):530–538CrossRefGoogle Scholar
  26. 26.
    Tang X, Liu K, Cui J, Wen F, Wang X (2012) Intentsearch: capturing user intention for one-click internet image search. IEEE Trans Pattern Anal Mach Intell 34(7):1342–1353CrossRefGoogle Scholar
  27. 27.
    Tian X, Tao D, Hua XS, Wu X (2010) Active reranking for web image search. IEEE Trans Image Process 19(3):805–820MathSciNetCrossRefGoogle Scholar
  28. 28.
    Tian X, Yang L, Wang J, Wu X, Hua XS (2011) Bayesian visual reranking. IEEE Trans Multimed 13(4):639–652CrossRefGoogle Scholar
  29. 29.
    The clipper library performs clipping, and offsetting of both lines and polygons (2014).
  30. 30.
    Uijlings JR, van de Sande KE, Gevers T, Smeulders AW (2013) Selective search for object recognition. Int J Comput Vis 104(2):154–171Google Scholar
  31. 31.
    Von Ahn L, Blum M, Hopper NJ, Langford J (2003) Captcha: Using hard ai problems for security. In: International conference on the theory and applications of cryptographic techniques. Springer, pp 294–311Google Scholar
  32. 32.
    Von Ahn L, Maurer B, McMillen C, Abraham D, Blum M (2008) recaptcha: human-based character recognition via web security measures. Science 321 (5895):1465–1468MathSciNetCrossRefGoogle Scholar
  33. 33.
    Wan S, Jin P, Yue L (2011) An effective image retrieval technique based on color perception. In: 2011 sixth international conference on image and graphics (ICIG). IEEE, pp 1017–1022Google Scholar
  34. 34.
    Yang L, Hanjalic A (2012) Prototype-based image search reranking. IEEE Trans Multimed 14(3):871–882CrossRefGoogle Scholar
  35. 35.
    Yang L, Hanjalic A (2013) Learning to rerank web images. IEEE MultiMedia 20(2):13–21Google Scholar
  36. 36.
    Yao T, Ngo CW, Mei T (2013) Circular reranking for visual search. IEEE Trans Image Process 22(4):1644–1655MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Wen-Yen Tseng
    • 1
  • Kai-Hsiang Chen
    • 2
    • 3
    Email author
  • Jen-Wei Huang
    • 1
  1. 1.Department of Electrical EngineeringNational Cheng Kung UniversityTainanTaiwan
  2. 2.Graduate Program of Multimedia Systems and Intelligent ComputingNational Cheng Kung UniversityTainanTaiwan
  3. 3.Graduate Program of Multimedia Systems and Intelligent ComputingAcademia SinicaTaipeiTaiwan

Personalised recommendations