Skip to main content
SpringerLink
Log in

What Matters for Chatbots? Analyzing Quality Measures for Facebook Messenger’s 100 Most Popular Chatbots

  • Conference paper
  • First Online:
Towards Integrated Web, Mobile, and IoT Technology

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 347))

Abstract

Chatbots are becoming mainstream. This work aims at ascertaining what are the enablers behind this popularity. To this end, we introduce four quality attributes, namely, “support of a minimal set of commands”, “foresee language variations”, “human-assistance provision” and “timeliness”. These criteria are applied to the 100 most popular Facebook Messenger chatbots. We review and measure both capacities and performance in order to find correlations between quality attribute fulfilment and popularity (chatbots’ ’likes’). Results show no significance correlations between quality attributes and chatbot popularity. However, the experiment comes up with three main contributions. First, a detailed description of how to measure these four quality attributes. Second, insights about how this assessment can be automatized, paving the way towards chatbot-evaluation platforms. Third, a checklist of frequently committed interaction errors as found in the revised chatbots. This might help developers to double-check their development.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Mobile-bots have the push-message ability, updating users with interesting news whenever they happen.

  2. 2.

    Web-bots were implemented ad-hoc, no standard UI was available for them to use, in contrast with mobile-bots running inside well-know apps, like Messenger, Telegram, Skype...

  3. 3.

    https://developers.facebook.com/docs/messenger-platform.

  4. 4.

    https://github.com/GoogleChrome/puppeteer.

  5. 5.

    https://github.com/petuscov/puppeteerFacebook.

References

  1. Beaver, L.: Chatbots are gaining traction (2017). https://www.businessinsider.com/chatbots-are-gaining-traction-2017-5. Business Insider

  2. Coniam, D.: The linguistic accuracy of chatbots: usability from an ESL perspective. Text Talk 34(5), 545–567 (2014)

    Article  Google Scholar 

  3. Emmet Connolly: Principles of bot design (2016). https://blog.intercom.com/principles-bot-design/. Inside Intercom

  4. Gartner: Chatbots will appeal to Modern Workers (2018). https://www.gartner.com/smarterwithgartner/chatbots-will-appeal-to-modern-workers/

  5. Klüwer, T.: From chatbots to dialog systems. Conversational agents and natural language interaction: techniques and effective practices, pp. 1–22 (2011)

    Google Scholar 

  6. Kuligowska, K.: Commercial chatbot: performance evaluation, usability metrics and quality standards of embodied conversational agents. Prof. Cent. Bus. Res. 2, 1–16 (2015)

    Google Scholar 

  7. Lester, J., Branting, K., Mott, B.: Conversational agents. In: The Practical Handbook of Internet Computing, pp. 220–240 (2004)

    Google Scholar 

  8. de Looper, C.: Facebook is now letting you pay for things through the Messenger Platform, September 2016. https://www.digitaltrends.com/social-media/facebook-messenger-chatbot-payments/

  9. Messina, C.: 2016 will be the year of conversational commerce, January 2016. https://medium.com/chris-messina/2016-will-be-the-year-of-conversational-commerce-1586e85e3991

  10. Morrissey, K., Kirakowski, J.: ‘Realness’ in chatbots: establishing quantifiable criteria. In: Kurosu, M. (ed.) HCI 2013. LNCS, vol. 8007, pp. 87–96. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39330-3_10

    Chapter  Google Scholar 

  11. Newlands, M.: Top 10 Practices for Making A Great Chatbot, June 2017. https://www.entrepreneur.com/article/296358

  12. Pauletto, S., et al.: Exploring expressivity and emotion with artificial voice and speech technologies. Logop. Phoniatr. Vocology 38(3), 115–125 (2013)

    Article  Google Scholar 

  13. Pereira, J., Díaz, Ó.: A quality analysis of Facebook Messenger’s most popular chatbots. In: Proceedings of the 33rd ACM/SIGAPP Symposium on Applied Computing, pp. 33:2144–2150 (2018)

    Google Scholar 

  14. Radziwill, N.M., Benton, M.C.: Evaluating Quality of Chatbots and Intelligent Conversational Agents. arXiv preprint arXiv:1704.04579 (2017)

  15. Ramos, R.: Screw the Turing test - chatbots don’t need to act human (2017). https://venturebeat.com/2017/02/03/screw-the-turing-test-chatbots-dont-need-to-act-human/

  16. Goh, O.S., Ardil, C., Wong, W., Fung, C.C.: A black-box approach for response quality evaluation of conversational agent systems. Int. J. Comput. Intell. Syst. 3, 37–41 (2012)

    Google Scholar 

  17. Shawar, B.A., Atwell, E.: Different measurements metrics to evaluate a chatbot system. In: Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies, pp. 89–96, April 2007

    Google Scholar 

  18. Sullivan, L.: Facebook Chatbots Hit 70% Failure Rate As Consumers Warm Up To The Tech (2017). https://www.mediapost.com/publications/article/295718/facebook-chatbots-hit-70-failure-rate-as-consumer.html

  19. Van Eeuwen, M., Van Der Kaap, H.: Mobile conversational commerce: messenger chatbots as the next interface between businesses and consumers (2017). http://essay.utwente.nl/71706/1/van Eeuwen_MA_BMS.pdf

  20. Wiggers, K.: Facebook Messenger can now find bots, order food, and more (2017). https://www.digitaltrends.com/social-media/facebook-messenger-news-f8-2017/

Download references

Author information

Authors and Affiliations

  1. ONEKIN Research Group, University of the Basque Country (UPV/EHU), Bilbao, Spain

    Juanan Pereira & Óscar Díaz

Authors
  1. Juanan Pereira
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Óscar Díaz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juanan Pereira .

Editor information

Editors and Affiliations

  1. Department of Information Systems, University of Agder, Kristiansand, Norway

    Tim A. Majchrzak

  2. ISISTAN-UNICEN-CONICET, Tandil, Argentina

    Cristian Mateos

  3. University of Bologna, Bologna, Italy

    Francesco Poggi

  4. Kristiania University College, Oslo, Norway

    Tor-Morten Grønli

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pereira, J., Díaz, Ó. (2019). What Matters for Chatbots? Analyzing Quality Measures for Facebook Messenger’s 100 Most Popular Chatbots. In: Majchrzak, T., Mateos, C., Poggi, F., Grønli, TM. (eds) Towards Integrated Web, Mobile, and IoT Technology. Lecture Notes in Business Information Processing, vol 347. Springer, Cham. https://doi.org/10.1007/978-3-030-28430-5_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-28430-5_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-28429-9

  • Online ISBN: 978-3-030-28430-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation