Vocally Specified Text Recognition in Natural Scenes for the Blind and Visually Impaired

  • Alhanouf AlnasserEmail author
  • Sharifa Al-Ghowinem
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 858)


Searching for items in our surrounding can be an instantaneous task for many. However, such task can consume much time and effort and cause frustration to the blind and the visually impaired. There is a great deal of information around us in textual form, such as signs and products’ labels that are not accessible by the blind. In most of the great efforts in assisting the blind and the visually impaired, a technology was introduced to identify hand-held objects by using a camera. It is not practical that the user must hold all items on a shelf until finding the desired item. The users need to search for what they want, not what the device or camera finds. This research is investigating how accurate could a developed application be in finding vocally specified text from natural scenes. To answer this research question, speech and text recognition modules can be combined to validate acquired data and measure the accuracy of locating vocally specified items. Enhancement techniques could be applied to individual modules to enhance the accuracy of combining both speech and recognition modules.


Image processing Text detection and recognition Speech recognition Artificial intelligence 



This work was supported by the Human Computer Interaction Research Group; Prince Sultan University, Riyadh, Saudi Arabia [RG-CCIS-2017-06-01].


  1. 1.
    Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R.C., Miller, R., Tatarowicz, A., White, S., White, B., Yeh, T.: VizWiz: nearly real-time answers to visual questions. In: Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology - UIST, p. 333. ACM Press, New York (2010).
  2. 2.
    Brady, E., Morris, M.R., Zhong, Y., White, S., Bigham, J.P.: Visual challenges in the everyday lives of blind people. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI 2013, p. 2117. ACM Press, New York (2013).
  3. 3.
    WHO, WHO | Visual impairment and blindness (2014).
  4. 4.
    World Health Organization, Universal eye health: a global action plan 2014–2019, p. 5 (2013).
  5. 5.
    Farhath, A., Amruthavarshini, K.R., Harshitha, S., Saranya, A., Velumadhavarao, R.: Development of shopping assistant using extraction of text images for visually impaired. In: 2014 Sixth International Conference on Advanced Computing (ICoAC), pp. 66–71. IEEE, December 2014.
  6. 6.
    Nanayakkara, S., Shilkrot, R., Yeo, K.P., Maes, P.: EyeRing. In: Proceedings of the 4th Augmented Human International Conference on - AH 2013, pp. 13–20. ACM Press, New York (2013).
  7. 7.
    Matusiak, K., Skulimowski, P., Strumillo, P.: Object recognition in a mobile phone application for visually impaired users. In: 2013 6th International Conference on Human System Interactions (HSI), pp. 479–484. IEEE, June 2013.
  8. 8.
    Bagwan, S.M.R., Sankpal, L.J.: VisualPal: a mobile app for object recognition for the visually impaired. In: 2015 International Conference on Computer, Communication and Control (IC4), pp. 1–6. IEEE, September 2015.
  9. 9.
    Bouazizi, I., Bouriss, F., Salih-Alj, Y.: Arabic reading machine for visually impaired people using TTS and OCR. In: 2013 4th International Conference on Intelligent Systems, Modelling and Simulation, pp. 225–229. IEEE, January 2013.
  10. 10.
    Yi, C., Tian, Y., Arditi, A.: Portable camera-based assistive text and product label reading from hand-held objects for blind persons. IEEE/ASME Trans. Mechatron. 19(3), 808–817 (2014). Scholar
  11. 11.
    Saleous, H., Shaikh, A., Gupta, R., Sagahyroon, A.: Read2Me: a cloud-based reading aid for the visually impaired. In: 2016 International Conference on Industrial Informatics and Computer Systems (CIICS), pp. 1–6. IEEE, March 2016.
  12. 12.
    Keefer, R., Bourbakis, N.: Interaction with a mobile reader for the visually impaired. In: 2009 21st IEEE International Conference on Tools with Artificial Intelligence, pp. 229–236. IEEE, May 2009
  13. 13.
    Lan, F., Zhai, G., Lin, W.: Lightweight smart glass system with audio aid for visually impaired people the smart glass system. In: TENCON 2015 - 2015 IEEE Region 10 Conference, pp. 4–7 (2015).
  14. 14.
    Munoz, R., Rong, X., Tian, Y.: Depth-aware indoor staircase detection and recognition for the visually impaired. In: 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 1–6. IEEE, July 2016.
  15. 15.
    Neto, L.B., Grijalva, F., Maike, V.R.M.L., Martini, L.C., Florencio, D., Baranauskas, M.C.C., Rocha, A., Goldenstein, S.: A kinect-based wearable face recognition system to aid visually impaired users. IEEE Trans. Human-Mach. Syst., 1–13 (2016).
  16. 16.
    Chincha, R., Tian, Y.: Finding objects for blind people based on SURF features. In: 2011 IEEE International Conference on Bioinformatics and Biomedicine Workshops (BIBMW), pp. 526–527. IEEE, November 2011.
  17. 17.
    Albraheem, L., AlDosari, R., AlKathiri, S., AlMotiry, H., Abahussain, H., AlHammad, L., Alshehri, M.: Third eye: an eye for the blind to identify objects using human-powered technology. In: 2015 International Conference on Cloud Computing (ICCC), pp. 1–6. IEEE, April 2015.
  18. 18.
    Khan, M.N.H., Arovi, M.A.H., Mahmud, H., Hasan, M.K., Rubaiyeat, H.A.: Speech based text correction tool for the visually impaired. In: 2015 18th International Conference on Computer and Information Technology (ICCIT), pp. 150–155. IEEE, December 2015.
  19. 19.
    Achkar, H.E., Zebian, M., Issa, J.: C-Thru: the intelligent device for autonomous visually impaired individuals. In: 2016 3rd International Conference on Advances in Computational Tools for Engineering Applications (ACTEA), pp. 241–244. IEEE, July 2016.
  20. 20.
    Bourbakis, N., Keefer, R., Dakopoulos, D., Esposito, A.: A multimodal interaction scheme between a blind user and the tyflos assistive prototype. In: 2008 20th IEEE International Conference on Tools with Artificial Intelligence, pp. 487–494. IEEE, November 2008.
  21. 21.
    Jayashree, D., Farhath, K.A., Amruthavarshini, R., Pavithra, S.: Voice based application as medicine spotter for visually impaired. In: 2016 Second International Conference on Science Technology Engineering and Management (ICONSTEM), pp. 56–60. IEEE, March 2016.
  22. 22.
    Sanketi, P.R., Coughlan, J.M.: Anti-blur feedback for visually impaired users of smartphone cameras. In: ASSETS. ACM Conference on Assistive Technologies, vol. 2010, pp. 233–234, October 2010.
  23. 23.
    Jayant, C., Ji, H., White, S., Bigham, J.P.: Supporting blind photography. In: The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility - ASSETS 2011, p. 203. ACM Press, New York (2011).
  24. 24.
    Alginahi, Y.: Preprocessing Techniques in Character Recognition. In: Mori, M. (ed.) Sciyo, Almadinah (2010). Scholar
  25. 25.
    Jung, K., In Kim, K., Jain, A.K.: Text information extraction in images and video: a survey. Pattern Recognit. 37(5), 977–997 (2004)Google Scholar
  26. 26.
    Feild, J.L., Learned-Miller, E.G.: Improving open-vocabulary scene text recognition. In: 2013 12th International Conference on Document Analysis and Recognition, pp. 604–608. IEEE, August 2013.
  27. 27.
  28. 28.
    Yu, D., Deng, L.: Automatic Speech Recognition A deep learning Approach. Springer, London (2015).

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Prince Sultan UniversityRiyadhSaudi Arabia

Personalised recommendations