iKnowU – Exploring the Potential of Multimodal AR Smart Glasses for the Decoding and Rehabilitation of Face Processing in Clinical Populations

  • Simon RuffieuxEmail author
  • Nicolas Ruffieux
  • Roberto Caldara
  • Denis Lalanne
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10515)


This article presents an explorative study with a smart glasses application developed to help visually impaired individuals to identify faces and facial expressions of emotion. The paper discusses three experiments in which different patients, suffering from distinct pathologies impairing vision, tested our application. These preliminary studies demonstrate the feasibility and usefulness of visual prostheses for face and emotion identification, and offer novel and interesting directions for future wearable see-through devices.


Wearable Smart glasses Face identification Emotion recognition Prosthesis Rehabilitation User-study 


  1. 1.
    Stevens, G.A., White, R.A., Flaxman, S.R., Price, H., Jonas, J.B., Keeffe, J., Leasher, J., Naidoo, K., Pesudovs, K., Resnikoff, S., Taylor, H., Bourne, R.R.A.: Global prevalence of vision impairment and blindness: magnitude and temporal trends, 1990-2010. Ophthalmology 120, 2377–2384 (2013)CrossRefGoogle Scholar
  2. 2.
    Spring, S.: Handicap visuel et cécité : évolution en suisse, pp. 5–22 (2012)Google Scholar
  3. 3.
    Sandnes, F.E.: What do low-vision users really want from smart glasses? Faces, text and perhaps no glasses at all. In: Miesenberger, K., Bühler, C., Penaz, P. (eds.) ICCHP 2016. LNCS, vol. 9758, pp. 187–194. Springer, Cham (2016). doi: 10.1007/978-3-319-41264-1_25 CrossRefGoogle Scholar
  4. 4.
    Cimarolli, V.R., Boerner, K., Brennan-Ing, M., Reinhardt, J.P., Horowitz, A.: Challenges faced by older adults with vision loss: a qualitative study with implications for rehabilitation. Clin. Rehabil. 26, 748–757 (2012)CrossRefGoogle Scholar
  5. 5.
    Bulling, A., Kunze, K.: Eyewear computers for human-computer interaction. Interactions 23, 70–73 (2016)CrossRefGoogle Scholar
  6. 6.
    Jafri, R., Ali, S.A.: Exploring the potential of eyewear-based wearable display devices for use by the visually impaired. In: Proceedings - 2014 3rd International Conference User Science Engineering Experience Engineer Engage, i-USEr 2014, pp. 119–124 (2015)Google Scholar
  7. 7.
    Terven, J.R., Salas, J., Raducanu, B.: New opportunities for computer vision-based assistive technology systems for the visually impaired. Comput. (Long. Beach. Calif) 47, 52–58 (2014)Google Scholar
  8. 8.
    Sujith, B., Safeeda, V.: Computer vision-based aid for the visually impaired persons - a survey and proposing. Int. J. Innov. Res. Comput. Commun. Eng. 365–370 (2014)Google Scholar
  9. 9.
    Jafri, R., Ali, S.A., Arabnia, H.R., Fatima, S.: Computer vision-based object recognition for the visually impaired in an indoors environment: a survey. Vis. Comput. 30, 1197–1222 (2014)CrossRefGoogle Scholar
  10. 10.
    Jafri, R.: A GPU-accelerated real-time contextual awareness application for the visually impaired on Google’s project Tango device. J. Supercomput. 73, 887–899 (2017)CrossRefGoogle Scholar
  11. 11.
    Mandal, B., Chia, S.-C., Li, L., Chandrasekhar, V., Tan, C., Lim, J.-H.: A wearable face recognition system on Google glass for assisting social interactions. In: Jawahar, C.V., Shan, S. (eds.) ACCV 2014. LNCS, vol. 9010, pp. 419–433. Springer, Cham (2015). doi: 10.1007/978-3-319-16634-6_31 Google Scholar
  12. 12.
    Chia, S., Mandal, B., Xu, Q., Li, L., Lim, J.: Enhancing social interaction with seamless face recognition on Google glass. In: Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct - MobileHCI 2015, pp. 750–757. ACM Press, New York (2015)Google Scholar
  13. 13.
    Xu, Q., Chia, S.C., Mandal, B., Li, L., Lim, J.-H., Mukawa, M.A., Tan, C.: SocioGlass: social interaction assistance with face recognition on Google glass. Sci. Phone Apps Mob. Devices 2, 7 (2016)Google Scholar
  14. 14.
    Hu, C., Zhai, G., Li, D.: An augmented-reality night vision enhancement application for see-through glasses. In: 2015 IEEE International Conference Multimedia Expo Work, ICMEW 2015 (2015)Google Scholar
  15. 15.
    Hwang, A.D., Peli, E.: An augmented-reality edge enhancement application for Google glass. Optom. Vis. Sci. 91, 1021–1030 (2014)CrossRefGoogle Scholar
  16. 16.
    Hicks, S.L., Wilson, I., Muhammed, L., Worsfold, J., Downes, S.M., Kennard, C.: A depth-based head-mounted visual display to aid navigation in partially sighted individuals. PLoS One 8, e67695 (2013)CrossRefGoogle Scholar
  17. 17.
    Oxsight: Oxsight company.
  18. 18.
    Krishna, S., Colbry, D., Black, J., Balasubramanian, V., Panchanathan, S.: A systematic requirements analysis and development of an assistive device to enhance the social interaction of people who are blind or visually impaired. In: Work Computer Vision Applications for Visually Impaired (2008)Google Scholar
  19. 19.
    Wagner, P.: Face recognition with opencv. OpenCV 2.4. 9.0 Doc (2012)Google Scholar
  20. 20.
    Mcduff, D.: AFFDEX SDK : a cross-platform real - time multi-face expression recognition toolkit. In: CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 3723–3726 (2016)Google Scholar
  21. 21.
    Rossion, B.: A network of occipito-temporal face-sensitive areas besides the right middle fusiform gyrus is necessary for normal face processing. Brain 126, 2381–2395 (2003)CrossRefGoogle Scholar
  22. 22.
    Caldara, R., Schyns, P., Mayer, E., Smith, M.L., Gosselin, F., Rossion, B.: Does prosopagnosia take the eyes out of face representations? Evidence for a defect in representing diagnostic facial information following brain damage. J. Cogn. Neurosci. 17, 1652–1666 (2005)CrossRefGoogle Scholar
  23. 23.
    Fiset, D., Blais, C., Royer, J., Richoz, A.-R., Dugas, G., Caldara, R.: Mapping the impairment in decoding static facial expressions of emotion in prosopagnosia. Soc. Cogn. Affect. Neurosci. (2017). (in press)Google Scholar
  24. 24.
    Ruffieux, N., Ramon, M., Lao, J., Colombo, F., Stacchi, L., Borruat, F.-X., Accolla, E., Annoni, J.-M., Caldara, R.: Residual perception of biological motion in cortical blindness. Neuropsychologia 93, 301–311 (2016)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2017

Authors and Affiliations

  • Simon Ruffieux
    • 1
    Email author
  • Nicolas Ruffieux
    • 2
  • Roberto Caldara
    • 2
  • Denis Lalanne
    • 3
  1. 1.HumanTechUniversity of Applied Sciences and ArtsFribourgSwitzerland
  2. 2.iBMLabUniversity of FribourgFribourgSwitzerland
  3. 3.Human-ISTUniversity of FribourgFribourgSwitzerland

Personalised recommendations