Living with Harmony: A Personal Companion System by Realbotix™

  • Kino CourseyEmail author
  • Susan Pirzchalski
  • Matt McMullen
  • Guile Lindroth
  • Yuri Furuushi


Existing personal assistants and agents are by design limited in their ability to form or encourage close personal bonds. The Harmony system is designed to be a customizable personal companion agent capable of close personal interaction via the user's phone, virtual reality headset, as well as through a physical interactive android body. In this chapter, we will describe the history that led to Harmony’s creation, the unique challenges and the overall system design. We will also look at user reactions to the system and anticipated future developments.


Androids Personal assistant Virtual reality Realbotix Embodied agent Companion agent 


  1. Alelo Develops an Interactive Robot for Learning Chinese. (2015). Retrieved March 16, 2015, from
  2. Apple’s Siri (n.d.). Retrieved from
  3. Becker-Asano, C., & Ishiguro, H. (2011). Evaluating facial displays of emotion for the android robot Geminoid F. In 2011 IEEE Workshop on, Affective Computational Intelligence (WACI) (pp. 1–8).Google Scholar
  4. Canepari, Z., Cooper, D., & Cott, E. Sex dolls that talk back. Robotica, episode 5 (New York Times Video, June 11, 2015).
  5. Clamp (Writer), Asaka, M. (Director), Gensho, T. (Producer), Ono, T. (Producer), & Sekido, Y. (Producer). (2002). Chobits [TV Series]. Japan: MadhouseGoogle Scholar
  6. Clarity Labs Lucida. Retrieved from
  7. Glas, D. F., Minato, T., Ishi, C. T., Kawahara, T., & Ishiguro, H. (2014). ERICA: The ERATO intelligent conversational android. In 2016 RO-MAN: The 25th IEEE International Symposium on Robot and Human Interactive Communication (pp. 22–29).Google Scholar
  8. Google’s Google Now. (n.d.). Retrieved from
  9. Hanson, D., Oh, J. H., Kim, W. S., Han, Y., Kim, J. Y., & Park, I. W. (2006). Design of android type humanoid robot Albert HUBO. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 1428–1433).Google Scholar
  10. Hanson, D., Olney, A., Prilliman, S., Mathews, E., Zielke, M., Hammons, D., … Stephanou, H. (2005). Upending the uncanny valley. In Proceedings of the National Conference on Artificial Intelligence (p. 1728)Google Scholar
  11. Harmon, A. (2010, July 5). Trying to forge a friendship with a robot named Bina 48. New York Times, p. A11. Retrieved from
  12. Hauswald, J., Laurenzano, M. A., Zhang, Y., Li, C., Rovinski A., Khurana, A., … Mars J. (2015). Sirius: An open end-to-end voice and vision personal assistant and its implications for future warehouse scale computers. In Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), ASPLOS’15 (pp. 223–238). New York, NY: ACMCrossRefGoogle Scholar
  13. Home Page of The Loebner Prize in Artificial Intelligence. Retrieved from
  14. Itti, L., Dhavale, N., & Pighin, F. (2003). Realistic avatar eye and head animation using a neurobiological model of visual attention. Proceedings of SPIE International Symposium on Optical Science and Technology, 5200, 64–78. Retrieved from Scholar
  15. Lee, S., Badler, J., & Badler, N. (2002). Eyes alive. ACM Transactions on Graphics, 21(3), 637–644. Retrieved from Scholar
  16. Liu, P., Glas, D. F., Kanda, T., Ishiguro, H., & Hagita N. (2014). How to train your robot - teaching service robots to reproduce human social behavior. In 2014 RO-MAN: The 23rd IEEE International Symposium on Robot and Human Interactive Communication (pp. 961–968).Google Scholar
  17. McMullen, M. (2014). U.S. Patent No. 8888553. Washington, DC: U.S. Patent and Trademark Office.Google Scholar
  18. Mori, M., MacDorman, K. F., & Kageki, N. (2012). The uncanny valley [from the field]. Robotics & Automation Magazine, IEEE, 19, 98–100.CrossRefGoogle Scholar
  19. Morph 3D. Retrieved from
  20. Nishio, H. I., & Hagita, N. (2007). Geminoid: Teleoperated android of an existing person. In A. C. D. P. Filho (Ed.), Humanoid robotics, new developments (pp. 343–352). London: IntechOpen Limited.Google Scholar
  21. Paro. Retrieved from
  22. Robokind. Retrieved from
  23. Samsung Bixby. Retrieved from Robokind
  24. Scheutz, M., & Arnold, T. (2016). Are we ready for sex robots? In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI) (pp. 351–358). Piscataway, NJ: IEEE. Retrieved from Scholar
  25. Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M.; Guez, A., … Hassabis, D. (2017, December 5). Mastering chess and shogi by self-play with a general reinforcement learning algorithm. Retrieved from
  26. Turing, A. M. (1950). I.—Computing machinery and intelligence. Mind, LIX(236), 433–460. Scholar
  27. Wikipedia contributors. (2018a, April 26). Roxxxy. In Wikipedia, The Free Encyclopedia. Retrieved 21:15, December 23, 2018, from
  28. Wikipedia contributors. (2018b, December 10). BINA48. In Wikipedia, The Free Encyclopedia. Retrieved 22:17, December 23, 2018, from
  29. Wikipedia contributors. (2018c, December 19). Her (film). In Wikipedia, The Free Encyclopedia. Retrieved 21:17, December 23, 2018, from
  30. Wikipedia contributors. (2018d, April 26). Eliza Effect. In Wikipedia, The Free Encyclopedia. Retrieved 23:26, November 13, 2018, from

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Kino Coursey
    • 1
    Email author
  • Susan Pirzchalski
    • 2
  • Matt McMullen
    • 3
  • Guile Lindroth
    • 4
  • Yuri Furuushi
    • 5
  1. 1.AI Research, Realbotix, Inc.DallasUSA
  2. 2.Hardware and System Engineering, Realbotix, Inc.DallasUSA
  3. 3.Realbotix, Inc.San MarcosUSA
  4. 4.Content and AI Research, Realbotix, Inc.CuritibaBrazil
  5. 5.Application Developement, Realbotix, Inc.CuritibaBrazil

Personalised recommendations