Skip to main content
SpringerLink
Log in

RGB-D Sensor-Based Computer Vision Assistive Technology for Visually Impaired Persons

  • Chapter
  • First Online:
Computer Vision and Machine Learning with RGB-D Sensors

Part of the book series: Advances in Computer Vision and Pattern Recognition ((ACVPR))

Abstract

A computer vision-based wayfinding and navigation aid can improve the mobility of blind and visually impaired people to travel independently. In this chapter, we focus on RGB-D sensor-based computer vision technologies in application to assist blind and visually impaired persons. We first briefly review the existing computer vision based assistive technology for the visually impaired. Then we provide a detailed description of the recent RGB-D sensor based assistive technology to help blind or visually impaired people. Next, we present the prototype system to detect and recognize stairs and pedestrian crosswalks based on RGB-D images. Since both stairs and pedestrian crosswalks are featured by a group of parallel lines, Hough transform is applied to extract the concurrent parallel lines based on the RGB (Red, Green, and Blue) channels. Then, the Depth channel is employed to recognize pedestrian crosswalks and stairs. The detected stairs are further identified as stairs going up (upstairs) and stairs going down (downstairs). The distance between the camera and stairs is also estimated for blind users. The detection and recognition results on our collected datasets demonstrate the effectiveness and efficiency of our developed prototype. We conclude the chapter by the discussion of the future directions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. 10 facts about blindness and visual impairment, World Health Organization (2009) Blindness and visual impairment (2009) www.who.int/features/factfiles/blindness/blindness_facts/en/index.html

  2. Advance Data Reports from the National Health Interview Survey (2008) http://www.cdc.gov/nchs/nhis/nhis_ad.htm

  3. Genensky SM, Baran P, Moshin H, Steingold H (1968) A closed circuit TV system for the visually handicapped

    Google Scholar 

  4. Kurzweil R (1976) The Kurzweil reading machine: a technical overview. Sci, Technol Handicapped, pp 3–11

    Google Scholar 

  5. Arditi A, Gillman A (1986) Computing for the blind user. Byte 11(3):199–211

    Google Scholar 

  6. Dakopoulos D, Bourbakis NG (2010) Wearable obstacle avoidance electronic travel aids for blind: a survey. IEEE Trans Syst Man Cybern Part C Appl Rev 40:25–35

    Article  Google Scholar 

  7. Giudice N, Legge GE (2008) Blind navigation and the role of technology. In: Helal A, Mokhtari M, Abdulrazak B (eds) The engineering handbook of smart technology for aging, disability, and independence. Wiley, Hoboken

    Google Scholar 

  8. Manduchi R, Coughlan J (2012) Computer vision without sight. Commun ACM 55(1):96–104

    Article  Google Scholar 

  9. Roentgen UR, Gelderblom GJ, Soede M, de Witte LP (2008) Inventory of electronic mobility aids for persons with visual impairments: a literature review. J Vis Impairment Blindness 102:702–724

    Google Scholar 

  10. Bonin-Font F, Ortiz A, Oliver G (2008) Visual navigation for mobile robots: a survey. J Intell Robot Syst 53(3):263–296

    Article  Google Scholar 

  11. DeSouza GN, Kak AC (2002) Vision for mobile robot navigation: a survey. IEEE Trans Pattern Anal Mach Intell 24(2):237–267

    Article  Google Scholar 

  12. Microsoft (2010) http://www.xbox.com/en-US/kinect

  13. PrimeSense http://www.primesense.com

  14. Bousbia-Salah M, Redjati A, Fezari M, Bettayeb M (2007) An ultrasonic navigation system for blind people. IEEE international conference on signal processing and communications (ICSPC), pp 1003–1006

    Google Scholar 

  15. Kao G (1996) FM sonar modeling for navigation. Department of Engineering Science, University of Oxford, Technical Report

    Google Scholar 

  16. Kuc R (2002) A sonar aid to enhance spatial perception of the blind: engineering design and evaluation. IEEE Trans Biomed Eng 49(10):1173–1180

    Article  Google Scholar 

  17. Laurent B, Christian T (2007) A sonar system modeled after spatial hearing and echolocating bats for blind mobility aid. Int J Phys Sci 2 (4):104–111

    Google Scholar 

  18. Morland C, Mountain D (2008) Design of a sonar system for visually impaired humans. The 14th international conference on auditory display, June 2008

    Google Scholar 

  19. Seeing with Sound—The vOICe:http://www.seeingwithdound.com

  20. The Argus II retinal prosthesis system. Available: http://www2-sight.eu/en/product-en

  21. VizWiz—Take a Picture, Speak a Question, and Get an Answer”. Available: http://www.vizwiz.org/

  22. Image Recognition APIs for photo albums and mobile commerce \({\vert }\) IQ Engines. Available: https://www.iqengines.com

  23. BrainPort lets you see with your tongue, might actually make it to market. Available: http://www.engadget.com/2009/08/14/brainport-lets-you-see-with-your-tongue-might-actually-make-it

  24. Chebat DR, Rainville C, Kupers R, Ptito M (2007) Tactile—visual acuity of the tongue in early blind individuals. Neuro Report 18(18):1901–1904

    Google Scholar 

  25. Williams MD, Ray CT, Griffith J, De l ’Aune W (2011) The use of a tactile-vision sensory substitution system as an augmentative tool for individuals with visual impairments. J. Vis Impairment Blindness, 105(1):45–50

    Google Scholar 

  26. Chebat DR, Schneider FC, Kupers R, Ptito M (2011) Navigation with a sensory substitution device in congenitally blind individuals. Neuro Report 22(7):342–347

    Google Scholar 

  27. Coughlan J, Shen HA (2006) Fast algorithm for finding crosswalks using figure-ground segmentation. The 2nd workshop on applications of computer vision, in conjunction with ECCV, 2006

    Google Scholar 

  28. Ivanchenko V, Coughlan J, Shen H (2008) Detecting and locating crosswalks using a camera phone. Computers Helping People with Special Needs. (Lecture notes in computer science), 5105:1122–1128

    Google Scholar 

  29. Advanyi R, Varga B, Karacs K (2010) Advanced crosswalk detection for the bionic eyeglass. In: Proceedings of the 12th international workshop on cellular nanoscale networks and their applications (CNNA) 2010, pp 1–5

    Google Scholar 

  30. Se S (2000) Zebra-crossing Detection for the Partially Sighted. IEEE Conf Comput Vis Pattern Recogn 2:211–217

    Google Scholar 

  31. Se S, Brady M (2000) Vision-based detection of stair-cases. In: Proceedings of fourth asian conference on computer vision (ACCV) 2000, pp 535–540

    Google Scholar 

  32. Uddin M, Shioyama T (2005) Bipolarity and projective invariant-based Zebra-crossing detection for the visually impaired. 1st IEEE Workshop on computer vision applications for the visually impaired, 2005

    Google Scholar 

  33. C Yi, Tian Y, (2011) Assistive text reading from complex background for blind persons. The 4th international workshop on camera-based document analysis and recognition (CBDAR), 2011

    Google Scholar 

  34. Wang S, Yi CY, Tian Y (2012) Signage detection and recognition for blind per-sons to access unfamiliar environments. J Comput Vis Image Process 2(2)

    Google Scholar 

  35. Yi C, Tian Y, Arditi A, (2013) Portable camera-based assistive text and product label reading from hand-held objects for blind persons. IEEE/ASME Trans Mechatron Accepted, 2013 http://dx.doi.org/10.1109/TMECH.2013.2261083

  36. Ye Z, Yi C, Tian Y (2013) Reading labels of cylinder objects for blind persons, IEEE international conference on multimedia & Expo (ICME), 2013

    Google Scholar 

  37. Yuan S, Tian Y, Arditi A (2011) Clothing matching for visually impaired persons. Technol Disabil 23:75–85

    Google Scholar 

  38. Hasanuzzaman F, Yang X, Tian Y (2012) Robust and effective component-based banknote recognition for the blind. IEEE Trans Syst Man Cybern Part C: Appl Rev, Jan 2012

    Google Scholar 

  39. Yang X, Yuan S, Tian Y, (2013) Assistive clothing pattern recognition for visually impaired people. IEEE Trans Hum Mach Syst. Accepted, 2013

    Google Scholar 

  40. Tian Y, Yang X, Yi C, Arditi A (2102) Toward a computer vision-based wayfinding aid for blind persons to access unfamiliar indoor environments. Mach Vis Appl, 2012

    Google Scholar 

  41. Pan H, Yi C, Tian Y (2103) A primary travelling assistant system of bus detection and recognition for visually impaired people. IEEE Workshop on multimodal and alternative perception for visually impaired people (MAP4VIP), in conjunction with ICME 2013

    Google Scholar 

  42. Joseph S, Zhang X, Dryanovski I, Xiao J, Yi C, Tian Y (2013) Semantic indoor navigation with a blind-user oriented augmented reality. IEEE International conference on systems, man, and cybernetics, 2013

    Google Scholar 

  43. Wang S, Pan H, Zhang C, Tian Y (2014) RGB-D image-based detection of stairs, pedestrian crosswalks and traffic signs. J Vis Commun Image Represent (JVCIR), 25:263–272, http://dx.doi.org/10.1016/j.jvcir.2013.11.005

  44. Arditi A Tian Y (2013) User interface preferences in the design of a camera-based navigation and wayfinding aid. J Vis Impairment Blindness, 107(2):118–129

    Google Scholar 

  45. Yi C, Tian Y (2011) Text string detection from natural scenes by structure-based partition and grouping. IEEE Trans Image Process 20(9):2594

    Google Scholar 

  46. Yi C, Tian Y (2012) Localizing text in scene images by boundary clustering, stroke segmentation, and string fragment classification. IEEE Trans Image Process 21(9):4256–4268

    Article  MathSciNet  Google Scholar 

  47. Yi C, Tian Y (2013) Text extraction from scene images by character appearance and structure modeling. Comput Vision Image Underst 117(2):182–194

    Google Scholar 

  48. Khan A, Moideen F, Khoo W, Zhu Z, Lopez J, KinDetect: kinect detecting objects. In: Proceedings of the 13th international conference on computers helping people with special needs, 7383, Miesenberger K, Karshmer A, Penaz P, Zagler W (2012) Springer, Berlin Heidelberg, Linz, Austria, July 11–13, 2012, 588–595

    Google Scholar 

  49. Ribeiro F, Florêncio D, Chou PA, Zhang Z, (2012) Auditory augmented reality: object sonification for the visually impaired. IEEE 14th international workshop on multimedia signal processing (MMSP), 2012

    Google Scholar 

  50. Tang H, Vincent M, Ro T, Zhu Z (2013) From RGB-D to low resolution tactile: smark sampling and early testing. IEEE workshop on multimodal and alternative perception for visually impaired people (MAP4VIP), in conjunction with ICME 2013

    Google Scholar 

  51. Wang Z, Liu H, Wang X, Qian Y (2014) Segment and label indoor scene Based on RGB-D for the visually impaired. MultiMedia modeling, Lecture Notes in Computer Science 8325:449–460

    Google Scholar 

  52. Lee YH, Medioni G (2011) A RGB-D camera based navigation for the visually impaired. RGB-D: advanced reasoning with depth camera work-shop, June 2011

    Google Scholar 

  53. Lee YH, Leung T, Médioni G (2012) Real-time staircase detection from a wearable stereo system. ICPR, Japan

    Google Scholar 

  54. Lu X, Manduchi, R (2005) Detection and localization of curbs and stairways using stereo vision ICRA, 2005

    Google Scholar 

  55. Wang S, Wang H (2009) 2D staircase detection using real adaboost. Information, communications and signal processing (ICICS), 2009

    Google Scholar 

  56. Chang C, Lin C (2001) LIBSVM: a Library for support vector machine, 2001. http://www.csie.ntu.edu.tw/~cjlin/libsvm

  57. Velazquez R (2010) Wearable assistive devices for the blind. Book chapter In Lay-Ekuakille A Mukhopadhyay SC (Eds), Wearable and autonomous biomedical devices and systems for smart environment: issues and characterization, LNEE 75, Springer, pp 331–349, 2010

    Google Scholar 

  58. Park CH, Howard AM (2013) Real-time haptic rendering and haptic telepresence robotic system for the visually impaired. World haptics conference (WHC), 2013

    Google Scholar 

  59. Tamjidi A, Ye C, Hong S (2013) 6-DOF pose estimation of a portable navigation aid for the visually impaired. IEEE international symposium on robotic and sensors environments, 2013

    Google Scholar 

  60. SR4000 User Manual (http://www.mesa-imaging)

  61. Marion AH, Michael AJ (Eds) (2008) Assistive technology for visually impaired and blind people. Springer, London

    Google Scholar 

  62. Zhang Z (2012) Microsoft kinect sensor and its effect. IEEE MultiMedia 19(2):4–10

    Google Scholar 

  63. Han J, Shao L, Xu D, Shotton J (2013) Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans Cybern, Oct 2013

    Google Scholar 

Download references

Acknowledgments

This work was supported in part by NSF Grant No. EFRI-1137172, IIP-1343402, FHWA DTFH61-12-H-00002, and Microsoft Research.

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, The City College of New York, New York, NY, 10031, USA

    Yingli Tian

Authors
  1. Yingli Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yingli Tian .

Editor information

Editors and Affiliations

  1. University of Sheffield, United Kingdom

    Ling Shao

  2. Civolution Technology, Eindhoven, The Netherlands

    Jungong Han

  3. Microsoft Research, Cambridge, United Kingdom

    Pushmeet Kohli

  4. Microsoft Research, Redmond, Washington, USA

    Zhengyou Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Tian, Y. (2014). RGB-D Sensor-Based Computer Vision Assistive Technology for Visually Impaired Persons. In: Shao, L., Han, J., Kohli, P., Zhang, Z. (eds) Computer Vision and Machine Learning with RGB-D Sensors. Advances in Computer Vision and Pattern Recognition. Springer, Cham. https://doi.org/10.1007/978-3-319-08651-4_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-08651-4_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-08650-7

  • Online ISBN: 978-3-319-08651-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation