Abstract
In this study we investigate the potential of cross-lingual video retrieval for how-to questions. How-to questions are the most frequent among wh-questions and constitute almost 1 % of the entire query stream. At the same time, how-to videos are popular on video sharing services. We analyzed a dataset of 500M+ Russian how-to questions. First, we carried out manual labelling of 1,000 queries that shows that about two thirds of all how-to question queries are potentially suitable for answers in the form of video in a language other than the language of the query. Then, we evaluated video retrieval quality for original and machine translated queries on a crowdsourcing platform. The evaluation reveals that machine translated questions yield video search quality comparable to the quality for original questions. Cross-lingual video search for how-to queries can improve recall and diversity of search results, as well as compensate the shortage of original content in emerging markets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
Weighted \(\kappa \) is a variant of \(\kappa \) that takes into account that the labels are interval variables: 3 is closer to 2 than it is to 1.
- 10.
We opted for DCG, since it also reflects how many relevant results were retrieved, not only how well the retrieved results were ranked (as in case of nDCG).
- 11.
References
Bojar, O., et al.: Findings of the 2015 workshop on statistical machine translation. In: WMT (2015)
Cao, J., Nunamaker, J.F.: Question answering on lecture videos: a multifaceted approach. In: JCDL (2004)
Chua, T.S., Hong, R., Li, G., Tang, J.: From text question-answering to multimedia QA on web-scale media resources. In: LS-MMRM Workshop (2009)
Filippova, K., Hall, K.B.: Improved video categorization from text metadata and user comments. In: SIGIR (2011)
Giampiccolo, D., et al.: Overview of the CLEF 2007 multilingual question answering track. In: Peters, C., et al. (eds.) CLEF 2007. LNCS, vol. 5152, pp. 200–236. Springer, Heidelberg (2008)
Hong, R., Wang, M., Li, G., Nie, L., Zha, Z.J., Chua, T.S.: Multimedia question answering. IEEE Trans. Multimed. 19(4), 72–78 (2012)
Li, G., et al.: Video reference: question answering on youtube. In: MM (2009)
Li, G., Li, H., Ming, Z., Hong, R., Tang, S., Chua, T.S.: Question answering over community-contributed web videos. IEEE Trans. Multimed. 17(4), 46–57 (2010)
Mitamura, T., et al.: Overview of the NTCIR-7 ACLIA tasks: advanced cross-lingual information access. In: NTCIR-7 Workshop (2008)
Nie, L., Wang, M., Gao, Y., et al.: Beyond text QA: multimedia answer generation by harvesting web information. IEEE Trans. Multimed. 15(2), 426–441 (2013)
Pang, B., Kumar, R.: Search in the lost sense of query: question formulation in web search queries and its temporal changes. In: ACL, vol. 2 (2011)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: ACL (2002)
Parton, K., McKeown, K.R., Allan, J., Henestroza, E.: Simultaneous multilingual search for translingual information retrieval. In: CIKM (2008)
Pavlick, E., Post, M., Irvine, A., Kachaev, D., Callison-Burch, C.: The language demographics of amazon mechanical turk. TACL 2, 79–92 (2014)
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., Makhoul, J.: A study of translation edit rate with targeted human annotation. In: AMTA (2006)
Spink, A., Ozmultu, H.C.: Characteristics of question format web queries: an exploratory study. Inf. Process. Manage. 38(4), 453–471 (2002)
Surdeanu, M., Ciaramita, M., Zaragoza, H.: Learning to rank answers to non-factoid questions from web collections. Comput. Linguist. 37(2), 351–383 (2011)
Torrey, C., Churchill, E.F., McDonald, D.W.: Learning how: the search for craft knowledge on the internet. In: CHI (2009)
Völske, M., Braslavski, P., Hagen, M., Lezina, G., Stein, B.: What users ask a search engine: analyzing one billion russian question queries. In: CIKM (2015)
Weber, I., Ukkonen, A., Gionis, A.: Answers, not links: Extracting tips from yahoo! answers to address how-to web queries. In: WSDM (2012)
Wu, Y.C., Chang, C.H., Lee, Y.S.: CLVQ: cross-language video question/answering system. In: IEEE MMSE (2004)
Yang, H., Chaisorn, L., Zhao, Y., Neo, S.Y., Chua, T.S.: VideoQA: question answering on news video. In: MM (2003)
Acknowledgments
We thank Yandex for preparing the dataset and granting access. Pavel Braslavski’s work was supported by RFBR, research project #14-07-00589-a. This publication was partially supported by the Dutch national program COMMIT (project P7 SWELL). We also thank all volunteers, who took part in the evaluation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Braslavski, P., Verberne, S., Talipov, R. (2016). Show Me How to Tie a Tie: Evaluation of Cross-Lingual Video Retrieval. In: Fuhr, N., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2016. Lecture Notes in Computer Science(), vol 9822. Springer, Cham. https://doi.org/10.1007/978-3-319-44564-9_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-44564-9_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-44563-2
Online ISBN: 978-3-319-44564-9
eBook Packages: Computer ScienceComputer Science (R0)