An Evaluation Methodology for Automatic Transcription System of Radiology Reports

  • Valéria Farinazzo Martins
  • Lincoln de Assis MouraJr.
Part of the Communications in Computer and Information Science book series (CCIS, volume 221)


This research combines knowledge from Computer Science and Health Science in order to propose an evaluation methodology for Automatic Transcription System of Radiology Reports. This methodology was designed based on Voice User interface requirements and specific requirements of automatic transcription systems of Radiology report. The same methodology was previously validated through some inspections and usability tests outside the hospital environment and, afterwards, it was used in two hospitals in São Paulo city. This approach aims to reduce costs of testing and available time by radiologists interviewed. Thus, the final product in this work consists of a set of criteria for evaluation of usability, comprising the name of the metric, evaluating method, steps to be followed and material to be used. By the use of this set, the evaluators can process the results of each requirement from the software.


Usability Evaluation Voice User Interface Automatic Transcription System of Radiology Reports Dictation System 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Allen, J., Perrault, C.: Analysing intention in utterances. Artificial Intelligence 15, 143–178 (1980)CrossRefGoogle Scholar
  2. 2.
    Kamm, C., Walker, M., Rabiner, L.: The role of speech processing in human computer intelligent communication. Speech Communication 23, 263–278 (1997)CrossRefGoogle Scholar
  3. 3.
    Price, P.: Evaluation of spoken language systems: the ATIS domain. In: Proceedings of the Third DARPA Speech and Natural Language Workshop. Morgan Kaufmann, San Francisco (1990)Google Scholar
  4. 4.
    Walker, M.A., Litman, D., Kamm, C., Abella, A.: Evaluating spoken dialogue agents with PARADISE: Two case studies. Computer Speech and Language 12(3), 317–347 (1998)CrossRefGoogle Scholar
  5. 5.
    Gibbon, D., Moore, R., Winski, R. (eds.): Hand-book of Standards and Resources for Spoken Language Systems. Mouton de Gruyter, New York (1997)Google Scholar
  6. 6.
    Dybkjaer, L., Bernsen, N.O., Dybkjaer, H.: Generality and objectivity: central issues in putting a dialogue evaluation tool into practical use. In: Hirschberg, J., Kamm, C., Walker, M. (eds.) Interactive Spoken Dialog Systems. Proceedings of a Workshop Sponsored by the Association for Computational Linguistics, Madrid, Spain, pp. 17–24. ACL (1997)Google Scholar
  7. 7.
    Nielsen, J.: Usability Engineering. Academic Press, Cambridge (1993)zbMATHGoogle Scholar
  8. 8.
    Sommerville, I.: Software Engineering, 6th edn. Addison Wesley, New York (2001)zbMATHGoogle Scholar
  9. 9.
    Kanal, K.M., Hangiandreou, N.J., Sykes, A.M.G., Eklund, H.E., Araoz, P.A., Leon, J.A., Erickson, B.J.: Initial Evaluation of a Continuous Speech Recognition Program for Radiology. Journal of Digital Imaging 14(1), 30–37 (2001)CrossRefGoogle Scholar
  10. 10.
    Kimberly, D.V.: A Methodology of Error Detection Improving Speech Recognition. Simon Fraser University Library, Canada (2006)Google Scholar
  11. 11.
    Voll, K., Atkins, S., Forster, B.: Improving the Utility of Speech Recognition Through Error Detection. Journal of Digital Imaging 21(4), 371–377 (2008)CrossRefGoogle Scholar
  12. 12.
    Paulett, J.M., Langlotz, C.P.: Improving language models for radiology speech recognition. Journal of Biomedical Informatics 42, 53–58 (2009)CrossRefGoogle Scholar
  13. 13.
    White, K.S.: Speech recognition implementation in radiology. Springer, Heidelberg (2005)Google Scholar
  14. 14.
    Bhan, S.N., Coblentz, C., Norman, G.R., Ali, A.H.: Effect of Voice Recognition on Radiologist Reporting Time. CARJ 59(4) (October 2008)Google Scholar
  15. 15.
    Durling, S., Lumsden, J.: Speech Recognition Use in Healthcare Applications. In: Proceedings of MoMM 2008, MoCoHe. ACM, New York (2008) 978-1-60558-269-6/08/0011Google Scholar
  16. 16.
    Diah, N.M., Ismaili, M., Ahmad, S., Dahari, M.K.M.: Usability testing for educational computer game using observation method. In: CAMP 2010, The First International Conference on Information Retrieval and Knowledge Management, Shah Alam, Malaysia (2010)Google Scholar
  17. 17.
    Mitchell, P.P.: A step-by-step guide to usability testing. iUniverse, Lincoln, NE (2007) Google Scholar
  18. 18.
    Möller, S.: A new taxonomy for the quality of telephone services based on spoken dialogue systems. In: Proc. 3rd SIGdial Worksh. on Discourse and Dialogue, US, Philadelphia, pp. 142–153 (2002)Google Scholar
  19. 19.
    Salvador, V.F.M., Oliveira Neto, J.S., Kawamoto A.L.: Requirement Engineering Contributions to Voice User Interface. In: First International Conference on Advances in Computer-Human Interaction, Sainte Luce, pp. 309–314 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Valéria Farinazzo Martins
    • 1
  • Lincoln de Assis MouraJr.
    • 2
  1. 1.Grupo de Processamento GráficoUniversidade Presbiteriana MackenzieSão PauloBrazil
  2. 2.Escola Politécnica da USP and Zilics eHealthSão PauloBrazil

Personalised recommendations