Computer-Based Readability Testing of Information Booklets for German Cancer Patients

  • Christian Keinki
  • Richard Zowalla
  • Monika Pobiruchin
  • Jutta Huebner
  • Martin Wiesner


Understandable health information is essential for treatment adherence and improved health outcomes. For readability testing, several instruments analyze the complexity of sentence structures, e.g., Flesch-Reading Ease (FRE) or Vienna-Formula (WSTF). Moreover, the vocabulary is of high relevance for readers. The aim of this study is to investigate the agreement of sentence structure and vocabulary-based (SVM) instruments. A total of 52 freely available German patient information booklets on cancer were collected from the Internet. The mean understandability level L was computed for 51 booklets. The resulting values of FRE, WSTF, and SVM were assessed pairwise for agreement with Bland–Altman plots and two-sided, paired t tests. For the pairwise comparison, the mean L values are LFRE = 6.81, LWSTF = 7.39, LSVM = 5.09. The sentence structure-based metrics gave significantly different scores (P < 0.001) for all assessed booklets, confirmed by the Bland–Altman analysis. The study findings suggest that vocabulary-based instruments cannot be interchanged with FRE/WSTF. However, both analytical aspects should be considered and checked by authors to linguistically refine texts with respect to the individual target group. Authors of health information can be supported by automated readability analysis. Health professionals can benefit by direct booklet comparisons allowing for time-effective selection of suitable booklets for patients.


Evidence-based health information Patient information booklets Readability Understandability Support vector machine 


Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants performed by any of the authors.


  1. 1.
    Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K (2011) Low health literacy and health outcomes: an updated systematic review. Ann Intern Med 155:97–107. CrossRefPubMedGoogle Scholar
  2. 2.
    Coulter A, Ellins J (2007) Effectiveness of strategies for informing, educating, and involving patients. BMJ 335:24–27. CrossRefPubMedPubMedCentralGoogle Scholar
  3. 3.
    Dirmaier J, Härter M (2012) Partizipative Entscheidungsfindung: Patientenbeteiligung bei Behandlungsentscheidungen in der medizinischen Versorgung [Shared decision making: patient involvement in treatment decisions in medical care]. BARMER GEK Gesundheitswesen Aktuell 212–235Google Scholar
  4. 4.
    Gorgojo L, Harris M, Garcia-Lopez E, Group CW (2012) National cancer control programmes: analysis of primary data from questionnaires Final Prelim Rep Eur Partnersh Action Cancer EPAACURL Httpwww Epaac Eunational-Cancer-Plans Accessed Novemb 2015Google Scholar
  5. 5.
    Bundesministerium für Gesundheit (2012) , Nationaler Krebsplan - Handlungsfelder, Ziele und Umsetzungsempfehlungen [National cancer plan—fields of action, objectives and implementation of recommendations]. Druckerei im Bundesministerium für Arbeit und Soziales, BerlinGoogle Scholar
  6. 6.
    Huebner J, Muenstedt K, Prott FJ, Stoll C, Micke O, Buentzel J, Muecke R, Senf B (2014) Online survey of patients with breast cancer on complementary and alternative medicine. Breast Care Basel Switz 9:60–63. CrossRefGoogle Scholar
  7. 7.
    Paul M, Davey B, Senf B, Stoll C, Münstedt K, Mücke R, Micke O, Prott FJ, Buentzel J, Hübner J (2013) Patients with advanced cancer and their usage of complementary and alternative medicine. J Cancer Res Clin Oncol 139:1515–1522. CrossRefPubMedGoogle Scholar
  8. 8.
    Keinki C, Seilacher E, Ebel M, Ruetters D, Kessler I, Stellamanns J, Rudolph I, Huebner J (2015) Information needs of cancer patients and perception of impact of the disease, of self-efficacy, and locus of control. J Cancer Educ Off J Am Assoc Cancer Educ 31:610–616. CrossRefGoogle Scholar
  9. 9.
    Huebner J, Micke O, Muecke R, Buentzel J, Prott FJ, Kleeberg U, Senf B, Muenstedt K, PRIO (Working Group Prevention and Integrative Oncology of the German Cancer Society) (2014) User rate of complementary and alternative medicine (CAM) of patients visiting a counseling facility for CAM of a German comprehensive cancer center. Anticancer Res 34:943–948PubMedGoogle Scholar
  10. 10.
    Flesch R (1948) A new readability yardstick. J Appl Psychol 32:221–233CrossRefPubMedGoogle Scholar
  11. 11.
    Friedman DB, Hoffman-Goetz L (2006) A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav off Publ Soc public Health Educ 33:352–373. CrossRefGoogle Scholar
  12. 12.
    Amstad T (1978) Wie verständlich sind unsere Zeitungen? [How readable are our newspapers?]. Doctoral thesis, Universität Zürich, SwitzerlandGoogle Scholar
  13. 13.
    Bamberger R, Vanacek (1984) Lesen-Verstehen-Lernen-Schreiben [Reading-Comprehension-Learning-Writing]. DiesterwegGoogle Scholar
  14. 14.
    Liebl P, Seilacher E, Koester M-J, Stellamanns J, Zell J, Hübner J (2015) What cancer patients find in the internet: the visibility of evidence-based patient information—analysis of information on German websites. Oncol Res Treat 38:212–218. CrossRefPubMedGoogle Scholar
  15. 15.
    Keinki C, Zowalla R, Wiesner M, Koester MJ, Huebner J (2016) Understandability of patient information booklets for patients with cancer. J Cancer Educ Off J Am Assoc Cancer Educ.
  16. 16.
    Robert Koch-Institut, die Gesellschaft der epidemiologischen Krebsregister in Deutschland e.V. (2013) Krebs in Deutschland 2009/2010 [Cancer in Germany 2009/2010]. Robert Koch-Institut, BerlinGoogle Scholar
  17. 17.
    Leroy G, Miller T, Rosemblat G, Browne A (2008) A balanced approach to health information evaluation: a vocabulary-based naïve Bayes classifier and readability formulas. J Am Soc Inf Sci Technol 59:1409–1419. CrossRefGoogle Scholar
  18. 18.
    Zowalla R, Wiesner M, Pfeifer D (2014) Automatically assessing the expert degree of online health content using SVMs. Stud Health Technol Inform 202:48–51PubMedGoogle Scholar
  19. 19.
    Joachims T (1998) Text categorization with support vector machines: learning with many relevant features. In: Nédellec C, Rouveirol C (eds) Machine learning: ECML-98. Springer Berlin Heidelberg, pp 137–142Google Scholar
  20. 20.
    Salton G, Buckley C (1988) Term-weighting approaches in automatic text retrieval. Inf Process Manag 24:513–523. CrossRefGoogle Scholar
  21. 21.
    Platt JC (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in large margin classifiers, MIT Press pp 61–74Google Scholar
  22. 22.
    Bland JM, Altman DG (1986) Statistical methods for assessing agreement between two methods of clinical measurement. Lancet Lond Engl 1:307–310CrossRefGoogle Scholar
  23. 23.
    Bland JM, Altman DG (1999) Measuring agreement in method comparison studies. Stat Methods Med Res 8:135–160. CrossRefPubMedGoogle Scholar
  24. 24.
    Hoefert H-W (2011) Wandel der Patientenrolle: neue Interaktionsformen im Gesundheitswesen [Change of the patient role: new forms of interaction in healthcare]. Hogrefe, GöttingenGoogle Scholar
  25. 25.
    Fagerlin A, Zikmund-Fisher BJ, Ubel PA (2011) Helping patients decide: ten steps to better risk communication. J Natl Cancer Inst 103:1436–1443. CrossRefPubMedPubMedCentralGoogle Scholar
  26. 26.
    Butow P, Brindle E, McConnell D, Boakes R, Tattersall M (1998) Information booklets about cancer: factors influencing patient satisfaction and utilization. Patient Educ Couns 33:129–141CrossRefPubMedGoogle Scholar
  27. 27.
    Cooley ME, Moriarty H, Berger MS, Selm-Orr D, Coyle B, Short T (1995) Patient literacy and the readability of written cancer educational materials. Oncol Nurs Forum 22:1345–1351PubMedGoogle Scholar
  28. 28.
    Garcia SF, Hahn EA, Jacobs EA (2010) Addressing low literacy and health literacy in clinical oncology practice. J Support Oncol 8:64–69PubMedPubMedCentralGoogle Scholar
  29. 29.
    Nicholls S, Hankins M, Hooley C, Smith H (2009) A survey of the quality and accuracy of information leaflets about skin cancer and sun-protective behaviour available from UK general practices and community pharmacies. J Eur Acad Dermatol Venereol JEADV 23:566–569. CrossRefPubMedGoogle Scholar
  30. 30.
    Singh J (2003) Reading grade level and readability of printed cancer education materials. Oncol Nurs Forum 30:867–870. CrossRefPubMedGoogle Scholar
  31. 31.
    Weintraub D, Maliski SL, Fink A, Choe S, Litwin MS (2004) Suitability of prostate cancer education materials: applying a standardized assessment tool to currently available materials. Patient Educ Couns 55:275–280. CrossRefPubMedGoogle Scholar
  32. 32.
    Hasan M, Kotov A, Carcone A et al (2016) A study of the effectiveness of machine learning methods for classification of clinical interview fragments into a large number of categories. J Biomed Inform 62:21–31. CrossRefPubMedPubMedCentralGoogle Scholar
  33. 33.
    Müller K (2001) Automatic detection of syllable boundaries combining the advantages of treebank and bracketed corpora training. In: Proceedings of the 39th annual meeting on Association for Computational Linguistics Association for Computational Linguistics, pp 410–417Google Scholar

Copyright information

© American Association for Cancer Education 2018

Authors and Affiliations

  1. 1.Department of Hematology and Medical OncologyUniversity Hospital JenaJenaGermany
  2. 2.Department of Medical InformaticsHeilbronn UniversityHeilbronnGermany
  3. 3.GECKO InstituteHeilbronn UniversityHeilbronnGermany

Personalised recommendations