Implementation of an eBook Reader System with the Features of Emotion Sensing and Robot Control

  • Jim-Min LinEmail author
  • Jan-Hwa Hsu
  • Zeng-Wei Hong
Conference paper
Part of the Smart Innovation, Systems and Technologies book series (SIST, volume 81)


The paper reports the implementation of an eBook reader with emotion sensing feature through Kinect sensor device. Recently, more and more traditional paper books have been transformed into an electronic form which is so called the electronic book (eBook). Currently, eBook contents could be in the form of combining various media through visual and audio communications. However, it still lacks the capability of real interactions between human and eBook to respond to user’s emotion as a person is reading eBooks. Therefore, we will apply emotional sensing technology to enhance the human-eBook interactions. So that an eBook reader device can sense a person’s reaction that reflects the situation and meaning as described in eBook contents. In this paper, the implementation of a novel eBook reader system integrating a Kinect sensor and the associate system development kit (SDK) will be reported. We will use a widely used face recognition technology, Active Appearance Model (AAM), for detecting user’s facial expression and emotions. With such a new feature, users may have funs and more realistic interactions with eBooks as a person is reading eBooks.


Human-Computer Interaction (HCI) Emotion sensing Facial expression detection e-book reader Kinect 



The authors would like to thank the Ministry of Science and Technology, Taiwan R.O.C. for the partly financial support (Grant numbers: MOST103-2221-E-035-051, MOST104-2221-E-035-008, and MOST105-2511-S-035-002-MY2) to this research.


  1. 1.
    Johal, W., Adam, C., Fiorino, H., Pesty, S., Jost, C., Duhaut, D.: Acceptability of a companion robot for children in daily life situations. In: IEEE Conference on Cognitive Infocommunications, Vietri sul Mare, pp. 31–36, (2014)Google Scholar
  2. 2.
    Zhang, S.-M., et al.: Research on the human computer interaction of E-learning. In: International Conference on Artificial Intelligence and Education (ICAIE) (2010)Google Scholar
  3. 3.
    Lee, K.M., Jung, Y., Kim, J., Kim, S.R.: Are physically embodied social agents better than disembodied social agents? The effects of physical embodiment, tactile interaction, and people’s loneliness in human–robot interaction. Int. J. Hum Comput Stud. 64, 962–973 (2006)CrossRefGoogle Scholar
  4. 4.
    Sakamoto, D., Kanda, T., Ono, T., Ishiguro, H., Hagita, N.: Android as a telecommunication medium with a human-like presence. In: presented at the Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction, Arlington, Virginia, USA (2007)Google Scholar
  5. 5.
    Chin, K.-Y.: IDML-based animated and physical pedagogical agents for computer-assisted learning. Ph.D. thesis, Department of Information Engineering and Computer Science, Feng Chia University, Taiwan (2011)Google Scholar
  6. 6.
    Li, K.-Y.: An IDML-based robot control mechanism. Master thesis, Department of Information Engineering and Computer Science, Feng Chia University, Taiwan (2009)Google Scholar
  7. 7.
    Manohar, V., Crandall, J.W.: Programming robots to express emotions: interaction paradigms, communication modalities, and context. IEEE Trans. Hum.-Mach. Syst. 44(3), 362–373 (2014)CrossRefGoogle Scholar
  8. 8.
    Lin, H.-A.: A study of the influences of e-book reading on tablet PC on the elementary school children’s reading ability and attitude. Master thesis, Graduate Institute of Library & Information Studies, National Taiwan Normal University, Taiwan (2011)Google Scholar
  9. 9.
    Hung, K.-H.: Product and marketing strategy research of ebooks and e-readers: based on Amazon, Barnes & Noble, and Apple. Master thesis, Department of Business Administration, National Taipei University, Taiwan (2010)Google Scholar
  10. 10.
    余宜芳、 葉靜瑜、 黃秀霜、 陳響亮, 「電子書教學互動教具之設計研究─以國小 五年級國語教材為例」, 2010 臺灣網際網路研討會大會 (TANET) (2010)Google Scholar
  11. 11.
    Zhang, Z.: Microsoft Kinect sensor and its effect. IEEE Multimedia Mag. 19(2), 4–10 (2012)CrossRefGoogle Scholar
  12. 12.
    Microsoft Kinect SDK, April 2014.
  13. 13.
    Microsoft Kinect SDK: Programming with the Kinect for Windows SDK, April 2014Google Scholar
  14. 14.
    Getting Started with the Kinect for Windows SDK Beta from Microsoft Research, Kinect for Windows SDK beta. Programming Guide, pp. 19–20, July 2011Google Scholar
  15. 15.
    Zhou, M., Liang, L., Sun, J., Wang, Y.-S.: AAM based face tracking with temporal matching and face segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 701–708 (2010)Google Scholar
  16. 16.
    Cootes, T., Edwards, G., Taylor, C.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)CrossRefGoogle Scholar
  17. 17.
    You, S.-D.: RDSL: a domain specific language for robot manipulation. Master thesis, Department of Information Engineering and Computer Science, Feng Chia University, Taiwan (2013)Google Scholar
  18. 18.
    XML Tutorial: Accessed July 2013
  19. 19.
    Lin, W.-J., Yueh, H.-P.: Examining college students’ reading behaviors and needs for Ebook readers. J. Libr. Inf. Stud., pp. 113–142, December 2012Google Scholar
  20. 20.
    Wikipedia, e-book:, December 2012
  21. 21.
    Lin, J.-M., Chiou, C.W., Lee, C.-Y., Hsiao, J.-R.: Supporting physical agents in an interactive e-book. In: The Ninth International Conference on Genetic and Evolutionary Computing (ICGEC 2015), pp. 243–252, Yangon, Myanmar 2015-08Google Scholar
  22. 22.
    Bluetooth Module:, July 2013
  23. 23.
    Rahman, A.S.Md.M., Alam, K.M., Saddik, A.El.: A prototype haptic EBook system to support immersive remote reading in a smart space. In: IEEE International Workshop on Haptic Audio Visual Environments and Games (HAVE), pp. 61–84 (2011)Google Scholar
  24. 24.
    Alam, K.M., Rahman, A.S.Md.M., Saddik, A.El.: HE-Book: a prototype haptic interface for immersive E-Book reading. In: World Haptics Conference (WHC), pp. 21–24 (2011)Google Scholar
  25. 25.
    Lin, C.-C.: Development of automatic recognition system for facial expression. Master thesis, Institute of Automation Technology, National Taipei University of Technology, Taiwan (2006)Google Scholar
  26. 26.
    Ekman, P.: Emotions Revealed Recognizing Faces and Feelings to Improve Communication and Emotional Life, 2nd edn. Holt Paperbacks, New York (2007). ISBN 080507516XGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Department of Information Engineering and Computer ScienceFeng Chia UniversityTaichung CityTaiwan
  2. 2.Department of Computer Science, Faculty of Information and Communication TechnologyUniversiti Tunku Abdul RahmanPetaling JayaMalaysia

Personalised recommendations