Abstract
Using one sensory modality to compensate for a modality that is unavailable is called Sensory Substitution and it is useful and often necessary for conveying some types of information effectively to people with disabilities. Using haptics to substitute for other modalities provides unique benefits as the tactile modality is incredibly flexible and underutilized. This chapter explores general purpose, media-focused, and interactive applications of Haptic Sensory Substitution as well as the future of Haptic Sensory Substitution and its implications for assistive technology.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., Amedi, A.: EyeMusic: Introducing a ‘visual’ colorful experience for the blind using auditory sensory substitution. Restorative Neurology and Neuroscience 32(2), 247–257 (2014)
Ando, B., Baglio, S., Marletta, V., Valastro, A.: A Haptic Solution to Assist Visually Impaired in Mobility Tasks. IEEE Transactions on Human-Machine Systems 45(5), 641–646 (2015)
Apple: Timecrest: The Door. https://apps.apple.com/za/app/timecrest-the-door/id1027546326 (2015), https://apps.apple.com/za/app/timecrest-the-door/id1027546326
Bach-Y-Rita, P., Collins, C.C., Saunders, F.A., White, B., Scadden, L.: Vision substitution by tactile image projection. Nature 221(5184), 963–964 (1969)
Bala, S., McDaniel, T., Panchanathan, S.: Visual-to-tactile mapping of facial movements for enriched social interactions. 2014 IEEE International Symposium on Haptic, Audio and Visual Environments and Games, HAVE 2014 – Proceedings pp. 82–87 (2014)
Benjamin, J.M.: The laser cane. Bulletin of prosthetics research pp. 443–50 (1974), http://www.ncbi.nlm.nih.gov/pubmed/4462934
Bliss, J.C., Katcher, M.H., Rogers, C.H., Shepard, R.P.: Optical-to-Tactile Image Conversion for the Blind. IEEE Transactions on Man-Machine Systems 11(1), 58–65 (1970)
Bucchieri, V.: Apparatus and method for presenting and controllably scrolling Braille text (2013), https://patents.google.com/patent/US8382480B2/en
Caspi, A., Dorn, J.D., McClure, K.H., Humayun, M.S., Greenberg, R.J., McMahon, M.J.: Feasibility study of a retinal prosthesis:Spatial vision with a 16-electrode implant. Archives of Ophthalmology 127(4), 398–401 (2009)
Cassinelli, A., Sampaio, E., Joffily, S.B., Lima, H.R., Gusmão, B.P.: Do blind people move more confidently with the Tactile Radar? Technology and Disability 26(2-3), 161–170 (2014)
Clary, P.: Lookout: an app to help blind and visually impaired people learn about their surroundings. https://www.blog.google/outreach-initiatives/accessibility/lookout-app-help-blind-and-visually-impaired-people-learn-about-their-surroundings/ (2018), https://www.blog.google/outreach-initiatives/accessibility/lookout-app-help-blind-and-visually-impaired-people-learn-about-their-surroundings/
Colwell, C., Petrie, H., Kornbrot, D., Hardwick, A., Furner, S.: Haptic virtual reality for blind computer users. In: Assets ’98 Proceedings of the third international ACM conference on Assistive technologies. pp. 92–99. Marina del Rey, California, USA (1998), https://dl.acm.org/citation.cfm?id=274515
De Felice, F., Renna, F., Attolico, G., Distante, A.: A haptic/acoustic application to allow blind the access to spatial information. Proceedings – Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, World Haptics 2007 pp. 310–315 (2007)
DMNagel: Timecrest: The Door. https://www.applevis.com/apps/ios/games/ timecrest-door (2017), https://www.applevis.com/apps/ios/games/timecrest-door
Dowino: A Blind Legend. https://play.google.com/store/apps/details?id=com.dowino.ABlindLegend (2019), https://play.google.com/store/apps/details?id=com.dowino.ABlindLegend&hl=en
Eagleman, D.: Plenary talks: A vibrotactile sensory substitution device for the deaf and profoundly hearing impaired. In: 2014 IEEE Haptics Symposium (HAPTICS). pp. xvii–xvii (2014), http://ieeexplore.ieee.org/document/6775419/
Ekstrom1, A.D.: Why vision is important to how we navigate. Hippocampus 73(4), 389–400 (2015)
Eskildsen, P., Morris, A., Collins, C.C., Bach-y Rita, P.: Simultaneous and successive cutaneous two-point thresholds for vibration. Psychonomic Science 14(4), 146–147 (1969)
Fakhri, B., Sharma, S., Soni, B., Chowdhury, A.: A Low Resolution Haptic Interface for Interactive Applications. HCI International pp. 1–6 (2019)
Fallis, A.: ’Smart’ Cane for the Visually Impaired: Design and Controlled Field Testing of an Affordable Obstacle Detection System. 12th International Conference on Mobility and Transport for Elderly and Disabled Persons 53(9), 1689–1699 (2010)
Freedom Scientific Inc.: Freedom Scientific Braille Displays and Keyboards. http://www.freedomscientific.com/ (2018), http://www.freedomscientific.com/
Geldard, F.A.: Cutaneous coding of optical signals: The optohapt. Perception & Psychophysics 1(11), 377–381 (1966)
Geldard, F.A., Sherrick, C.E.: The cutaneous “rabbit”: A perceptual illusion. Science 178(4057), 178–179 (1972)
GHARIEB, W., NAGIB, G.: Smart Cane for Blinds. Proc. 9th Int. Conf. on AI Applications (August), 253–262 (2015), c:{%}5CUsers{%}5Cjessica{%}5CBIBLIOTECA{%}5Cdesign{%}5CMetodologiaexperimental-GuiBonsiepe.pdf{%}5Cnhttp://www.researchgate.net/profile/Gihan{\_}Nagib/publication/255615346{\_}Smart{\_}Cane{\_}for{\_}Blinds/links/542020190cf241a65a1b065a.pdf{%}5Cnhttp://www.researchgate.net/publi
He, L., Wan, Z., Findlater, L., Froehlich, J.E.: TacTILE: A Preliminary Toolchain for Creating Accessible Graphics with 3D-Printed Overlays and Auditory Annotations. Proc. 19th Int. ACM SIGACCESS Conf. Comput. Access. pp. 397–398 (2017), https://doi.org/10.1145/3132525.3134818
Homer Jacobson: The Informational Capacity of the Human Ear. Science 112(2901), 143–144 (1950), http://science.sciencemag.org/content/112/2901/143
Ikei, Y., Wakamatsu, K., Fukuda, S.: Vibratory tactile display of image-based textures. IEEE Computer Graphics and Applications 17(6), 53–61 (1997)
Jacobson, H.: The informational capacity of the human eye. Science 113(2933), 292–293 (1951)
Jansson, G., Petrie, H., Colwell, C., Kornbrot, D.: Haptic virtual environments for blind people: Exploratory experiments with two devices. The International Journal of Virtual Reality 3(4), 8–17 (1999), https://pdfs.semanticscholar.org/348e/45107167a0325051e60c883c153572a127e4.pdf%0Ahttp://www.ijvr.org/issues/pre/4-1/2.pdf
Karyn, G., Gina, M., Alessandra, M., Robert, C., Peter, B., Graeme, C.: A comparison of Tactaid II+ and Tactaid 7 use by adults with a profound hearing impairment. Ear and Hearing 20(6), 471–482 (1999), http://www.scopus.com/inward/record.url?eid=2-s2.0-0033436316&partnerID=40&md5=1058dec9323d2a378c6fcb685db9acc5
König, H., Schneider, J., Strothotte, T.: Haptic Exploration of Virtual Buildings Using Non-Realistic Haptic Rendering. Society pp. 377–384 (2000)
König, H., Schneider, J., Strothotte, T.: Orientation and Navigation in Virtual Haptic-Only Environments. In: Paelke, V., Volbracht, S. (eds.) Proceedings User Guidance in Virtual Environments. pp. 123–136. Shaker Verlag, Aachen, Germany (2001)
Krishna, S., Bala, S., McDaniel, T., McGuire, S., Panchanathan, S.: VibroGlove: an assistive technology aid for conveying facial expressions. In: CHI ’10 Extended Abstracts on Human Factors in Computing Systems. pp. 3637–3642 (2010), https://doi.org/10.1145/1753846.1754031
Kuc, R.: Binaural sonar electronic travel aid provides vibrotactile cues for landmark, reflector motion and surface texture classification. IEEE Transactions on Biomedical Engineering 49(10), 1173–1180 (2002)
Lahav, O., Mioduser, D.: Construction of cognitive maps of unknown spaces using a multi-sensory virtual environment for people who are blind. Computers in Human Behavior 24(3), 1139–1155 (2008)
Landau, S., Wells, L.: Merging Tactile Sensory Input and Audio Data by Means of The Talking Tactile Tablet. Proc. Eurographics’03 2(60), 414–418 (2003)
Lévesque, V., Pasquero, J., Hayward, V., Legault, M.: Display of virtual braille dots by lateral skin deformation: feasibility study. ACM Transactions on Applied Perception 2(2), 132–149 (2005)
Li, Z., Snavely, N.: MegaDepth: Learning Single-View Depth Prediction from Internet Photos. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition pp. 2041–2050 (2018)
Liu, C., Wu, J., Furukawa, Y.: FloorNet: A unified framework for floorplan reconstruction from 3D scans. In: European Conference on Computer Vision. vol. 11210 LNCS, pp. 203–219 (2018)
Loomis, J.M.: Tactile letter recognition under different modes of stimulus presentation. Perception & Psychophysics 16(2), 401–408 (1974)
Maidenbaum, S., Levy-Tzedek, S., Chebat, D.R., Amedi, A.: Increasing accessibility to the blind of virtual environments, using a virtual mobility aid based on the “EyeCane”: Feasibility study. PLoS ONE 8(8) (2013)
Mascetti, S., Bernareggi, C., Belotti, M.: TypeInBraille: A Braille-based Typing Application for Touchscreen Devices. In: The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility. pp. 295–296. Dundee, Scotland, UK (2011)
McDaniel, T., Krishna, S., Balasubramanian, V., Colbry, D., Panchanathan, S.: Using a haptic belt to convey non-verbal communication cues during social interactions to individuals who are blind. HAVE 2008 – IEEE International Workshop on Haptic Audio Visual Environments and Games Proceedings (October), 13–18 (2008)
McDaniel, T., Villanueva, D., Krishna, S., Panchanathan, S.: MOVeMENT: A framework for systematically mapping vibrotactile stimulations to fundamental body movements. HAVE 2010 – 2010 IEEE International Symposium on Haptic Audio-Visual Environments and Games, Proceedings pp. 13–18 (2010)
McDaniel, T.L., Krishna, S., Colbry, D., Panchanathan, S.: Using tactile rhythm to convey interpersonal distances to individuals who are blind. CHI Extended Abstracts pp. 4669–4674 (2009), https://dl.acm.org/citation.cfm?id=1520718
Meijer, P.B.: An Experimental System for Auditory Image Representations. IEEE Transactions on Biomedical Engineering 39(2), 112–121 (1992)
Menikdiwela, M.P., Dharmasena, K.M., Abeykoon, A.M.S.: Haptic based walking stick for visually impaired people. 2013 International Conference on Circuits, Controls and Communications, CCUBE 2013 pp. 1–6 (2013)
Merzenich, M.M., Michelson, R.P., Pettit, C.R., Schindler, R.A., Reid, M.: Neural Encoding of Sound Sensation Evoked by Electrical Stimulation of the Acoustic Nerve. Annals of Otology, Rhinology & Laryngology 82(4), 486–503 (1973)
Metz, C.: Facebook’s AI Is Now Automatically Writing Photo Captions. https://www.wired.com/2016/04/facebook-using-ai-write-photo-captions-blind-users/ (2016), https://www.wired.com/2016/04/facebook-using-ai-write-photo-captions-blind-users/
Metz, R.: BLITAB. https://www.technologyreview.com/s/603336/this-500-tablet-brings-words-to-blind-users-fingertips/ (2017), https://www.technologyreview.com/s/603336/this-500-tablet-brings-words-to-blind-users-fingertips/
Meyer, I., Mikesch, H.: FEER the Game of Running Blind. http://www.feer.at/index.php/en/home/ (2018), http://www.feer.at/index.php/en/home/
Microsoft: Seeing AI. https://www.microsoft.com/en-us/ai/seeing-ai (2018), https://www.microsoft.com/en-us/ai/seeing-ai
Miller, I., Pather, A., Milbury, J., Hathy, L., O’Day, A., Spence, D.: Guidelines and Standards for Tactile Graphics, 2010. http://www.brailleauthority.org/tg/web-manual/index.html (2011), http://www.brailleauthority.org/tg/web-manual/index.html
Miyazaki, M., Hirashima, M., Nozaki, D.: The “Cutaneous Rabbit” Hopping out of the Body. Journal of Neuroscience 30(5), 1856–1860 (2010), http://www.jneurosci.org/cgi/doi/10.1523/JNEUROSCI.3887-09.2010
Monacelli, A.M., Cushman, L.A., Kavcic, V., Duffy, C.J.: Spatial disorientation in Alzheimer’s disease: The remembrance of things passed. Neurology 61(11), 1491–1497 (2003), https://pdfs.semanticscholar.org/e957/14321d7fb821b421f2897496ccd1d10fed60.pdf
National Federation of the Blind Jernigan Institute: The Braille Literacy Crisis in America, Facing the Truth, Reversing the Trend, Empowering the Blind. Tech. rep., National Federation of the Blind, Baltimore Maryland (2009)
Nau, A., Bach, M., Fisher, C.: Clinical Tests of Ultra-Low Vision Used to Evaluate Rudimentary Visual Perceptions Enabled by the BrainPort Vision Device. Translational Vision Science & Technology 2(3), 1 (2013), http://tvst.arvojournals.org/Article.aspx?doi=10.1167/tvst.2.3.1
Novich, S.D., Eagleman, D.M.: Using space and time to encode vibrotactile information: toward an estimate of the skin’s achievable throughput. Experimental Brain Research 233(10), 2777–2788 (2015)
Novich, S.D.: Sound-to-Touch Sensory Substitution and Beyond (2015), https://scholarship.rice.edu/handle/1911/88379
Panchanathan, S., Chakraborty, S., McDaniel, T.: Social Interaction Assistant: A Person-Centered Approach to Enrich Social Interactions for Individuals with Visual Impairments. IEEE Journal on Selected Topics in Signal Processing 10(5), 942–951 (2016)
Pasquero, J.: Survey on communication through touch. McGill Centre for Intelligent Machines 6(August), 1–28 (2006), http://scholar.google.com/scholar?hl=en{&}btnG=Search{\&}q=intitle:Survey+on+Communication+through+Touch{\#}0
Pietrzak, T., Pecci, I., Martin, B.: Static and dynamic tactile directional cues experiments with VTPlayer mouse. In: Proceedings of the Eurohaptics conference. pp. 63–68. Paris, France (2006)
Régo, N.: The Game of Running Blind in FEAR. https://coolblindtech.com/the-game-of-running-blind-in-fear/ (2018), https://coolblindtech.com/the-game-of-running-blind-in-fear/
Riener, a., Hartl, H.: Personal Radar: a self-governed support system to enhance environmental perception. Proceedings of BCS-HCI 2012 pp. 147–156 (1974), http://dl.acm.org/citation.cfm?id=2377933{%}5Cnpapers://c80d98e4-9a96-4487-8d06-8e1acc780d86/Paper/p15116
Bach-y Rita, P., Danilov, Y., Tyler, M., Grimm, R.J.: Late human brain plasticity: vestibular substitution with a tongue BrainPort human-machine interface. Plasticidad y Restauracion Neurologica 4(1-2), 31–34 (2005), http://www.medigraphic.com/pdfs/plasticidad/prn-2005/prn051_2f.pdf%5Cnhttp://www.ncbi.nlm.nih.gov/pubmed/15194608%0Ahttp://doi.wiley.com/10.1196/annals.1305.006
Rosen, S.: Long Cane Techniques Study Guide Step-By-Step A Guide to Mobility Techniques. https://tech.aph.org/sbs/04_sbs_lc_study.html#4, https://tech.aph.org/sbs/04_sbs_lc_study.html#4
Rosenthal, J., Edwards, N., Villanueva, D., Krishna, S., McDaniel, T., Panchanathan, S.: Design, implementation, and case study of a pragmatic vibrotactile belt. In: IEEE Transactions on Instrumentation and Measurement. vol. 60, pp. 114–125 (2011)
Sampaio, E., Maris, S., Bach-y Rita, P.: Brain plasticity: ‘Visual’ acuity of blind persons via the tongue. Brain Research 908(2), 204–207 (2001)
Schmidt, R.N., Lisy, F.J., Prince, T.S., Shaw, G.S.: Refreshable braille display system (1998), https://patents.google.com/patent/US6354839B1/en
Semwal, S.: MoVE: Mobiltiy training in haptic virtual environment. Tech. rep., University of Colorado at Colorado Springs, Colorado Springs (2001), https://pdfs.semanticscholar.org/243e/b3d64990f34d0b126b36132acc17e4f50737.pdf{%}0Ahttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.24.6646{\&}rep=rep1{\&}type=pdf
Sharkey, P., Sik Lanyi, C., Standen, P., University of Reading. ICDVRAT, D.o.C.: Multisensory virtual environment for supporting blind persons’ acquisition of spatial cognitive mapping, orientation, and mobility skills (1993), 279 (2002)
Smithmaitrie, P., Kanjantoe, J., Tandayya, P.: Touching force response of the piezoelectric Braille cell. Disability and Rehabilitation: Assistive Technology 3(6), 360–365 (2008)
Stageberg, S.: The Device That Refreshes: How to Buy a Braille Display. https://www.afb.org/aw/5/6/14669 (2004), https://www.afb.org/aw/5/6/14669
Sweller, J., Ayres, P., Kalyuga, S.: Amassing information: The information store principle. In: Cognitive Load Theory (2011)
Tan, H.Z., Pentland, A.: Tactual. Disptays for Wearabte Computing. Personal Technologies pp. 225–230 (1997)
Troxel, D.: Experiments in Tactile and Visual Reading. IEEE Transactions on Human Factors in Electronics 8(4), 261–263 (1967), https://ieeexplore.ieee.org/stamp/stamp.jsp?tp={&}arnumber=1698280
Tzovaras, D., Moustakas, K., Nikolakis, G., Strintzis, M.G.: Interactive mixed reality white cane simulation for the training of the blind and the visually impaired. Personal and Ubiquitous Computing 13(1), 51–58 (2009)
Van Erp, J., Self, B.: Tactile Displays for Orientation, Navigation and Communication in Air, Sea and Land Environments, vol. 323 (2008)
Wall, S., Brewster, S.: Feeling What You Hear: Tactile Feedback for Navigation of Audio Graphs. In: CHI 2006 Proceedings, Disabilities. pp. 1123–1132 (2006)
Wall, S.A., Brewster, S.: Sensory substitution using tactile pin arrays: Human factors, technology and applications. Signal Processing 86(12), 3674–3695 (2006)
Wang, Q., Levesque, V., Pasquero, J., Hayward, V.: A haptic memory game using the STRESS 2 tactile display. In: Proceedings of CHI 2006. p. 271 (2006)
Ward, J., Meijer, P.: Visual experiences in the blind induced by an auditory sensory substitution device. Consciousness and Cognition 19(1), 492–500 (2010)
White, B.W., Saunders, F.A., Scadden, L., Bach-Y-Rita, P., Collins, C.C.: Seeing with the skin. Perception & Psychophysics 7(1), 23–27 (1970)
WikiHow: How to Swing a Golf Club. https://www.wikihow.com/Swing-a-Golf-Club (2019), https://www.wikihow.com/Swing-a-Golf-Club
Yanagida, Y., Kakita, M., Lindeman, R.W., Kume, Y., Tetsutani, N.: Vibrotactile letter reading using a low-resolution tactor array. Proceedings – 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, HAPTICS pp. 400–406 (2004)
Yang, U., Jang, Y., Kim, G.J.: Designing a VibroTactile Wear for Close Range Interaction for VRbased Motion Training. Icat 2002 pp. 4–9 (2002), http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.103.5793
Zhao, Y., Bennett, C.L., Benko, H., Cutrell, E., Holz, C., Morris, M.R., Sinclair, M.: Enabling People with Visual Impairments to Navigate Virtual Reality with a Haptic and Auditory Cane Simulation. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems – CHI ’18 pp. 1–14 (2018), http://dl.acm.org/citation.cfm?doid=3173574.3173690
Acknowledgements
The authors thank Arizona State University and the National Science Foundation for their funding support. The preparation of this chapter is supported by the National Science Foundation under Grant No. 1828010 and 1069125.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Fakhri, B., Panchanathan, S. (2020). Haptics for Sensory Substitution. In: McDaniel, T., Panchanathan, S. (eds) Haptic Interfaces for Accessibility, Health, and Enhanced Quality of Life. Springer, Cham. https://doi.org/10.1007/978-3-030-34230-2_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-34230-2_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-34229-6
Online ISBN: 978-3-030-34230-2
eBook Packages: Computer ScienceComputer Science (R0)