Abstract
We consider the problem of reused answer retrieval for community question answering (CQA): given a question q, retrieve answers \(a_{i}^{j}\) posted in response to other questions \(q_{i} (\ne q)\), where \(a_{i}^{j}\) serves as an answer to q. While previous work evaluated this task by manually annotating the relationship between q and \(a_{i}^{j}\), this approach does not scale for large-scale CQA sites. We therefore explore an automatic evaluation method for reused answer retrieval, which computes nDCG by defining the gain value of each retrieved answer as a ROUGE score that treats the original answers to q as gold summaries. Our answer retrieval experiment suggests that effective reused answer retrieval systems may not be the same as effective gold answer retrieval systems. We provide case studies to discuss the benefits and limitations of our approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This approach is similar to the classical residual collection method for evaluating relevance feedback in IRÂ [3].
References
Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M.: Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv:1606.01847 (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
Hull, D.: Using statistical testing in the evaluation of retrieval experiments. In: Proceedings of ACM SIGIR 1993, pp. 329–338 (1993)
Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. (TOIS) 20(4), 422–446 (2002)
Lin, C.Y.: Rouge: a package for automatic evaluation of summaries. Text Summarization Branches Out (2004)
Nakov, P., et al.: SemEval-2017 task 3: community question answering. In: Proceedings of SemEval-2017, pp. 27–48 (2017)
Saito, K., Shin, A., Ushiku, Y., Harada, T.: Dualnet: domain-invariant network for visual question answering. In: 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 829–834. IEEE (2017)
Sakai, T.: Metrics, statistics, tests. In: Ferro, N. (ed.) PROMISE 2013. LNCS, vol. 8173, pp. 116–163. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-54798-0_6
Sakai, T.: Unanimity-aware gain for highly subjective assessments. Proc. EVIA 2017, 39–42 (2017)
Tamaki, K., Togashi, R., Fujita, S., Kato, S., Maeda, H., Sakai, T.: Classifying community qa questions that contain an image. In: ICTIR (2018)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, HW., Fujita, S., Sakai, T. (2020). Towards Automatic Evaluation of Reused Answers in Community Question Answering. In: Wang, F., et al. Information Retrieval Technology. AIRS 2019. Lecture Notes in Computer Science(), vol 12004. Springer, Cham. https://doi.org/10.1007/978-3-030-42835-8_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-42835-8_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-42834-1
Online ISBN: 978-3-030-42835-8
eBook Packages: Computer ScienceComputer Science (R0)