Speech Communication Through the Skin: Design of Learning Protocols and Initial Findings

  • Jaehong Jung
  • Yang Jiao
  • Frederico M. Severgnini
  • Hong Z. TanEmail author
  • Charlotte M. Reed
  • Ali Israr
  • Frances Lau
  • Freddy Abnousi
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10919)


Evidence for successful communication through the sense of touch is provided by the natural methods of tactual communication that have been used for many years by persons with profound auditory and visual impairments. However, there are still currently no wearable devices that can support speech communication effectively without extensive learning. The present study reports the design and testing of learning protocols with a system that translates English phonemes to haptic stimulation patterns (haptic symbols). In one pilot study and two experiments, six participants went through the learning and testing of phonemes and words that involved different vocabulary sizes and learning protocols. We found that with a distinctive set of haptic symbols, it was possible for the participants to learn phonemes and words in small chunks of time. Further, our results provided evidence of the memory consolidation theory in that recognition performance improved after a period of inactivity on the part of a participant. Our findings pave the way for future work on improving the haptic symbols and on protocols that support the learning of a tactile speech communication system in hours as opposed to the much longer periods of time that are required to learn a new language.


Haptic speech communication Memory consolidation Phoneme 



This work was partially supported by a research grant from Facebook Inc. The authors thank all participants for their time and dedication, and Keith Klumb for project management.


  1. 1.
    Reed, C.M., Rabinowitz, W.M., Durlach, N.I., Braida, L.D., Conway-Fithian, S., Schultz, M.C.: Research on the Tadoma method of speech communication. J. Acoust. Soc. Am. 77, 247–257 (1985)CrossRefGoogle Scholar
  2. 2.
    Tan, H.Z., Durlach, N.I., Reed, C.M., Rabinowitz, W.M.: Information transmission with a multifinger tactual display. Percept. Psychophys. 61(6), 993–1008 (1999)CrossRefGoogle Scholar
  3. 3.
    Reed, C.M., Durlach, N.I.: Note on information transfer rates in human communication. Pres. Teleoper. Virtual Environ. 7(5), 509–518 (1998)CrossRefGoogle Scholar
  4. 4.
    Reed, C.M., Durlach, N.I., Braida, L.D.: Research on tactile communication of speech: a review. In: American Speech-Language-Hearing Association (ASHA) Monographs Number 20, May 1982, 23 p. (1982)Google Scholar
  5. 5.
    Reed, C.M., Durlach, N.I., Delhorne, L.A., Rabinowitz, W.M., Grant, K.W.: Research on tactual communication of speech: ideas, issues, and findings. Volta Rev. 91(5), 65–78 (1989). Monograph entitled “Research on the use of sensory aids for hearing-impaired people,” N.S. McGarr, EditorGoogle Scholar
  6. 6.
    Kirman, J.H.: Tactile communication of speech: a review and analysis. Psychol. Bull. 80, 54–74 (1973)CrossRefGoogle Scholar
  7. 7.
    Yuan, H., Reed, C.M., Durlach, N.I.: Tactual display of consonant voicing as a supplement to lipreading. J. Acoust. Soc. Am. 118(2), 1003–1015 (2005)CrossRefGoogle Scholar
  8. 8.
    Israr, A., Reed, C.M., Tan, H.Z.: Discrimination of vowels with a multi-finger tactual display. In: Proceedings of the Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 17–24 (2008)Google Scholar
  9. 9.
    Novich, S.D., Eagleman, D.M.: Using space and time to encode vibrotactile information: toward an estimate of the skin’s achievable throughput. Exp. Brain Res. 233, 2777–2788 (2015)CrossRefGoogle Scholar
  10. 10.
    Verrillo, R.T., Gescheider, G.A.: Perception via the sense of touch (Chap. 1). In: Summers, I.R. (ed.) Tactile Aids for the Hearing Impaired, pp. 1–36. Whurr Publishers, London (1992)Google Scholar
  11. 11.
    Tan, H.Z., Durlach, N.I., Rabinowitz, W.M., Reed, C.M., Santos, J.R.: Reception of Morse code through motional, vibrotactile, and auditory stimulation. Percept. Psychophys. 59(7), 1004–1017 (1997)CrossRefGoogle Scholar
  12. 12.
    Wade, T., Holt, L.L.: Incidental categorization of spectrally complex non-invariant auditory stimuli in a computer game task. J. Acoust. Soc. Am. 118(4), 2618–2633 (2005)CrossRefGoogle Scholar
  13. 13.
    Karni, A., Dudai, Y., Born, J.: The consolidation and transformation of memory. Neuron 88, 20–32 (2015)CrossRefGoogle Scholar
  14. 14.
    Ecroyd, D.H.: Voice and Articulation: A Hanbook, pp. 63–87. Scott Foresman & Co Publisher, Glenview (1966)Google Scholar
  15. 15.
    Reed, C.M., et al.: A phonemic-based tactual display for speech communication. IEEE Trans. Haptics (2018, in preparation)Google Scholar
  16. 16.
    Levitt, H.: Transformed up-down methods in psychoacoustics. J. Acoust. Soc. Am. 49(2B), 467–477 (1971)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Jaehong Jung
    • 1
  • Yang Jiao
    • 1
  • Frederico M. Severgnini
    • 1
  • Hong Z. Tan
    • 1
    Email author
  • Charlotte M. Reed
    • 2
  • Ali Israr
    • 3
  • Frances Lau
    • 3
  • Freddy Abnousi
    • 3
  1. 1.Purdue UniversityWest LafayetteUSA
  2. 2.Massachusetts Institute of TechnologyCambridgeUSA
  3. 3.Facebook Inc.Menlo ParkUSA

Personalised recommendations