Skip to main content
SpringerLink
Log in

Boosting a Rule-Based Chatbot Using Statistics and User Satisfaction Ratings

  • Conference paper
  • First Online:
Book cover Artificial Intelligence and Natural Language (AINL 2017)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 789))

Included in the following conference series:

Abstract

Using data from user-chatbot conversations where users have rated the answers as good or bad, we propose a more efficient alternative to a chatbot’s keyword-based answer retrieval heuristic. We test two neural network approaches to the near-duplicate question detection task as a first step towards a better answer retrieval method. A convolutional neural network architecture gives promising results on this difficult task.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Accorsi, P., Patel, N., Lopez, C., Panckhurst, R., Roche, M.: Seek & hide: anonymising a french sms corpus using natural language processing techniques. Lingvisticæ Investigationes 35(2), 163–180 (2012)

    Google Scholar 

  2. Afzal, N., Wang, Y., Liu, H.: MayoNLP at SemEval-2016 Task 1: semantic textual similarity based on lexical semantic net and deep learning semantic model. In: Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, 16–17 June 2016, pp. 674–679 (2016)

    Google Scholar 

  3. Baumeister, R.F., Bratslavsky, E., Finkenauer, C., Vohs, K.D.: Bad is stronger than good. Rev. Gen. Psychol. 5(4), 323 (2001)

    Article  Google Scholar 

  4. Bernhard, D., Gurevych, I.: Answering learners’ questions by retrieving question paraphrases from social Q&A sites. In: Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications, pp. 44–52. ACL (2008)

    Google Scholar 

  5. Bikel, D.M., Schwartz, R., Weischedel, R.M.: An algorithm that learns what’s in a name. Mach. Learn. 34(1), 211–231 (1999)

    Article  Google Scholar 

  6. Bogdanova, D., dos Santos, C.N., Barbosa, L., Zadrozny, B.: Detecting semantically equivalent questions in online user forums. In: Proceedings of the 19th Conference on Computational Natural Language Learning, CoNLL 2015, Beijing, China, 30–31 July 2015, pp. 123–131 (2015)

    Google Scholar 

  7. Denis, P., Sagot, B.: Coupling an annotated corpus and a lexicon for state-of-the-art pos tagging. Lang. Resour. Evaluation 46(4), 721–736 (2012)

    Article  Google Scholar 

  8. Feng, M., Xiang, B., Glass, M.R., Wang, L., Zhou, B.: Applying deep learning to answer selection: a study and an open task. In: 2015 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2015, Scottsdale, AZ, USA, 13–17 December 2015, pp. 813–820 (2015)

    Google Scholar 

  9. Goldberg, Y.: Neural Network Methods for Natural Language Processing. Morgan & Claypool, San Rafael (2017)

    Google Scholar 

  10. Higashinaka, R., Minami, Y., Dohsaka, K., Meguro, T.: Issues in predicting user satisfaction transitions in dialogues: individual differences, evaluation criteria, and prediction models. In: Lee, G.G., Mariani, J., Minker, W., Nakamura, S. (eds.) IWSDS 2010. LNCS (LNAI), vol. 6392, pp. 48–60. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-16202-2_5

    Chapter  Google Scholar 

  11. Hogan, D., Leveling, J., Wang, H., Ferguson, P., Gurrin, C.: Dcu@fire 2011: SMS-based FAQ retrieval. In: 3rd Workshop of the Forum for Information Retrieval Evaluation, FIRE, pp. 2–4 (2011)

    Google Scholar 

  12. Hone, K.S., Graham, R.: Subjective assessment of speech-system interface usability. In: INTERSPEECH, pp. 2083–2086 (2001)

    Google Scholar 

  13. Jalbert, N., Weimer, W.: Automated duplicate detection for bug tracking systems. In: IEEE International Conference on Dependable Systems and Networks with FTCS and DCC (DSN 2008), pp. 52–61. IEEE (2008)

    Google Scholar 

  14. Jeon, J., Croft, W.B., Lee, J.H.: Finding semantically similar questions based on their answers. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 617–618. ACM (2005)

    Google Scholar 

  15. Jijkoun, V., de Rijke, M.: Retrieving answers from frequently asked questions pages on the web. In: Proceedings of the 14th ACM International Conference on Information and Knowledge Management, pp. 76–83. ACM (2005)

    Google Scholar 

  16. Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)

  17. Kim, Y.: Convolutional neural networks for sentence classification. In: EMNLP, pp. 1746–1751. ACL (2014)

    Google Scholar 

  18. Liu, C.W., Lowe, R., Serban, I.V., Noseworthy, M., Charlin, L., Pineau, J.: How not to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023 (2016)

  19. Lowe, R., Serban, I.V., Noseworthy, M., Charlin, L., Pineau, J.: On the evaluation of dialogue systems with next utterance classification. arXiv preprint arXiv:1605.05414 (2016)

  20. Malakasiotis, P., Androutsopoulos, I.: Learning textual entailment using SVMs and string similarity measures. In: Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 42–47. ACL (2007)

    Google Scholar 

  21. Manning, C.D., Raghavan, P., Schütze, H., et al.: Introduction to Information Retrieval, vol. 1. Cambridge University Press, Cambridge (2008)

    Book  Google Scholar 

  22. Muthmann, K., Petrova, A.: An automatic approach for identifying topical near-duplicate relations between questions from social media Q/A sites. In: Proceeding of WSDM 2014 Workshop: Web-Scale Classification: Classifying Big Data from the Web (2014)

    Google Scholar 

  23. Reitter, D., Moore, J.D.: Predicting success in dialogue. In: Proceedings of the 45th Annual Meeting of the ACL, ACL 2007, 23–30 June 2007, Prague, Czech Republic (2007)

    Google Scholar 

  24. Ritter, A., Cherry, C., Dolan, W.B.: Data-driven response generation in social media. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 583–593. ACL (2011)

    Google Scholar 

  25. Rodrigues, J.A., Saedi, C., Maraev, V., Silva, J., Branco, A.: Ways of asking and replying in duplicate question detection. In: Ide, N., Herbelot, A., Màrquez, L. (eds.) Proceedings of the 6th Joint Conference on Lexical and Computational Semantics, *SEM @ACM 2017, Vancouver, Canada, 3–4 August 2017, pp. 262–270. Association for Computational Linguistics (2017). https://doi.org/10.18653/v1/S17-1030

  26. Seddah, D., Sagot, B., Candito, M., Mouilleron, V., Combet, V.: The French Social Media Bank: a treebank of noisy user generated content. In: 24th International Conference on Computational Linguistics, COLING 2012 (2012)

    Google Scholar 

  27. Vinyals, O., Le, Q.: A neural conversational model. arXiv preprint arXiv:1506.05869 (2015)

  28. Walker, M., Langkilde, I., Wright, J., Gorin, A., Litman, D.: Learning to predict problematic situations in a spoken dialogue system: experiments with how may I help you? In: Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, pp. 210–217. Association for Computational Linguistics (2000)

    Google Scholar 

  29. Walker, M.A., Litman, D.J., Kamm, C.A., Abella, A.: PARADISE: A framework for evaluating spoken dialogue agents. In: Proceedings of the Eighth Conference on European Chapter of the Association for Computational Linguistics, pp. 271–280. ACL (1997)

    Google Scholar 

  30. Wu, Y., Zhang, Q., Huang, X.: Efficient near-duplicate detection for Q&A forum. In: Fifth International Joint Conference on Natural Language Processing, IJCNLP 2011, Chiang Mai, Thailand, 8–13 November 2011, pp. 1001–1009 (2011)

    Google Scholar 

  31. Xue, X., Jeon, J., Croft, W.B.: Retrieval models for question and answer archives. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 475–482. ACM (2008)

    Google Scholar 

  32. Yin, W., Schütze, H., Xiang, B., Zhou, B.: ABCNN: attention-based convolutional neural network for modeling sentence pairs. arXiv preprint arXiv:1512.05193 (2015)

Download references

Acknowledgements

This research is partly funded by the Regional Council of Brittany through an ARED grant. The present research was also partly supported by the CLARIN and ANI/3279/2016 grants. We are grateful to Telsi for providing the data.

Author information

Authors and Affiliations

  1. LIDILE EA3874, University of Rennes 2, Rennes, France

    Octavia Efraim

  2. CLASP, University of Gothenburg, Gothenburg, Sweden

    Vladislav Maraev

  3. Department of Informatics, Faculty of Sciences, University of Lisbon, Lisbon, Portugal

    João Rodrigues

Authors
  1. Octavia Efraim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vladislav Maraev
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. João Rodrigues
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vladislav Maraev .

Editor information

Editors and Affiliations

  1. ITMO University, St. Petersburg, Russia

    Andrey Filchenkov

  2. University of Helsinki, Helsinki, Finland

    Lidia Pivovarova

  3. Mendel University , Brno, Czech Republic

    Jan Žižka

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Efraim, O., Maraev, V., Rodrigues, J. (2018). Boosting a Rule-Based Chatbot Using Statistics and User Satisfaction Ratings. In: Filchenkov, A., Pivovarova, L., Žižka, J. (eds) Artificial Intelligence and Natural Language. AINL 2017. Communications in Computer and Information Science, vol 789. Springer, Cham. https://doi.org/10.1007/978-3-319-71746-3_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-71746-3_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-71745-6

  • Online ISBN: 978-3-319-71746-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation