Synthetic Speech in Therapy of Auditory Hallucinations

  • Kamil Sorokosz
  • Izabela Stefaniak
  • Artur JanickiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10415)


In this article we propose using speech synthesis in the therapy of auditory verbal hallucinations, which are sometimes called “voices”. During a therapeutic session a patient converses with an avatar, which is controlled by a therapist. The avatar, based on the XFace model and commercial text-to-speech systems, uses a high quality synthetic voice synchronized with lip movements. A proof-of-concept is demonstrated, as well as the results of preliminary experiments with six patients. The initial results are highly encouraging – all the patients claimed that the therapy helped them, and they also highly assessed the quality of the avatar’s speech and its synchronization with the animations.


Speech synthesis Auditory hallucinations Assistive technologies Avatar Visual speech 


  1. 1.
    eSpeak - a free TTS engine.
  2. 2.
    Balcí, K.: Xface: open source toolkit for creating 3D faces of an embodied conversational agent. In: Butz, A., Fisher, B., Krüger, A., Olivier, P. (eds.) SG 2005. LNCS, vol. 3638, pp. 263–266. Springer, Heidelberg (2005). doi: 10.1007/11536482_25
  3. 3.
    Bilikiewicz, A., Pużyński, S., Rybakowski, J., Wciórka, J.: Psychiatry. Wydawnictwo Medyczne Urban & Partner, Wrocław II (2002)Google Scholar
  4. 4.
    Brinkman, W.P., Hartanto, D., Kang, N., de Vliegher, D., Kampmann, I.L., Morina, N., Emmelkamp, P.G.M., Neerincx, M.: A virtual reality dialogue system for the treatment of social phobia. In: Extended Abstracts on Human Factors in Computing Systems, CHI 2012, pp. 1099–1102. ACM, New York (2012)Google Scholar
  5. 5.
    Buchanan, R.W., Kreyenbuhl, J., Kelly, D.L., Noel, J.M., Boggs, D.L., Fischer, B.A., Himelhoch, S., Fang, B., Peterson, E., Aquino, P.R., et al.: The 2009 schizophrenia PORT psychopharmacological treatment recommendations and summary statements. Schizophrenia Bull. 36(1), 71–93 (2010)CrossRefGoogle Scholar
  6. 6.
    Chadwick, P., Birchwood, M.: The omnipotence of voices. A cognitive approach to auditory hallucinations. Br. J. Psychiatry 164(2), 190–201 (1994)CrossRefGoogle Scholar
  7. 7.
    Craig, T.K.J., Rus-Calafell, M., Ward, T., Fornells-Ambrojo, M., McCrone, P., Emsley, R., Garety, P.: The effects of an audio visual assisted therapy aid for refractory auditory hallucinations (avatar therapy): study protocol for a randomised controlled trial. Trials 16(1), 349 (2015)CrossRefGoogle Scholar
  8. 8.
    Creer, S., Cunningham, P.G.S., Yamagishi, J.: Building personalized synthetic voices for individuals with Dysarthria using the HTS toolkit. In: Mullenix, J., Stern, S. (eds.) Computer Synthesized Speech Technologies: Tools for Aiding Impairment, pp. 92–115. IGI Global press, Hershey (2010)CrossRefGoogle Scholar
  9. 9.
    Ellis, D.: Time-frequency automatic gain control (2010). https://labrosaeecolumbiaedu/matlab/tf_agc
  10. 10.
    Falconer, C.J., Rovira, A., King, J.A., Gilbert, P., Antley, A., Fearon, P., Ralph, N., Slater, M., Brewin, C.R.: Embodying self-compassion within virtual reality and its effects on patients with depression. Br. J. Psychiatry 2(1), 74–80 (2016)Google Scholar
  11. 11.
    Freitas, D., Kouroupetroglou, G.: Electronic speech processing for persons with disabilities. Technol. Disabil. 20, 135–156 (2008)Google Scholar
  12. 12.
    Grogan, S., Conner, M., Willits, D., Norman, P.: Development of a questionnaire to measure patients’ satisfaction with general practitioners’ services. Br. J. Gen. Pract. 45(399), 525–529 (1995)Google Scholar
  13. 13.
    Huckvale, M., Leff, J., Williams, G.: Avatar therapy: an audio-visual dialogue system for treating auditory hallucinations. In: Proceedings Interspeech 2013, pp. 392–396, August 2013Google Scholar
  14. 14.
    Janicki, A., Bloch, J., Taylor, K.: Visual speech synthesis for Polish using keyframe based animation. In: Pułka, A., Golonek, T. (eds.) Proceedings of International Conference on Signals and Electronics Systems, ICSES 2010, pp. 423–426. IEEE, September 2010Google Scholar
  15. 15.
    Jarema, M.: Psychiatry. In: PZWL (2016). (in Polish)Google Scholar
  16. 16.
    Larøi, F., Sommer, I.E., Blom, J.D., Fernyhough, C., Hugdahl, K., Johns, L.C., McCarthy-Jones, S., Preti, A., Raballo, A., Slotema, C.W., et al.: The characteristic features of auditory verbal hallucinations in clinical and nonclinical groups: state-of-the-art overview and future directions. Schizophrenia Bull. 38(4), 724–733 (2012)CrossRefGoogle Scholar
  17. 17.
    Pagliari, C., Burton, C., Mckinstry, B.H., Wolters, M.: Psychosocial implications of avatar use in supporting therapy for depression. Stud. Health Technol. Inform. 181, 329–333 (2012)Google Scholar
  18. 18.
    Paulo, S., Oliveira, L.C., Mendes, C., Figueira, L., Cassaca, R., Viana, C., Moniz, H.: DIXI – a generic text-to-speech system for European Portuguese. In: Teixeira, A., Lima, V.L.S., Oliveira, L.C., Quaresma, P. (eds.) PROPOR 2008. LNCS, vol. 5190, pp. 91–100. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-85980-2_10 CrossRefGoogle Scholar
  19. 19.
    Sakoe, H., Chiba, S.: Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans. Acoust. Speech Signal Process. 26(1), 43–49 (1978)CrossRefzbMATHGoogle Scholar
  20. 20.
    Sarver, N.W., Beidel, D., Spitalnick, J.S.: The feasibility and acceptability of virtual environments in the treatment of childhood social anxiety disorder. J. Clin. Child Adolesc. Psychol. 43, 63–73 (2013)CrossRefGoogle Scholar
  21. 21.
    Stahl, S.M.: Stahl’s Essential Psychopharmacology: Neuroscientific Basis and Practical Applications. Cambridge University Press, Cambridge (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Kamil Sorokosz
    • 1
  • Izabela Stefaniak
    • 2
  • Artur Janicki
    • 1
    Email author
  1. 1.Institute of TelecommunicationsWarsaw University of TechnologyWarsawPoland
  2. 2.Institute of Psychiatry and NeurologyWarsawPoland

Personalised recommendations