Supportive Care in Cancer

, Volume 26, Issue 1, pp 129–138 | Cite as

A Delphi study to develop indicators of cancer patient experience for quality improvement

  • Kathryn E. WilliamsEmail author
  • Janet Sansoni
  • Darcy Morris
  • Cristina Thompson
Original Article



The purpose of this study was to develop prioritised indicators to measure cancer patient experience and thus guide quality improvement in the delivery of patient care.


A Delphi study, consisting of two surveys and three workshops, was employed to gather expert opinions on the most important indicators to measure. Survey participants were 149 health professionals, academics/technical experts and consumers. The first survey was based on a literature review which identified 105 elements of care within 14 domains of patient experience. These were rated on a 7-point Likert scale, with ‘1’ representing high importance. Elements with mean ratings between 1.0 and 2.0 were retained for the second survey. The 43 least-important elements were omitted, four elements were revised and nine new elements added. Consensus was defined as at least 70% of participants rating an element ‘1’ or ‘2’. Multivariate and cluster analyses were used to develop 20 draft indicators, which were presented to 51 experts to refine and prioritise at the three workshops.


All elements in the second survey were rated ‘1’ or ‘2’ by 81% of participants. Workshop participants agreed strongly on the four most important indicators: coordinated care, access to care, timeliness of the first treatment, and communication. Other indicators considered highly important were follow-up care for survivors; timeliness of diagnosis; information relating to side effects, pain and medication; comprehensibility of information provided to patients; and needs assessment.


Experts identified priorities with a high level of consensus, providing a rigorous foundation for developing prioritised indicators of quality in cancer patient experience.


Patient experience Quality indicators Quality improvement Delphi Cancer 



Funding for the current research was provided by the Cancer Institute of New South Wales. The authors wish to thank Nicole Cook and Margaret Barbouttis of the Cancer Institute NSW and the many experts who contributed their time to this study

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflicts of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.


  1. 1.
    Daly L (2012) Developing a quality cancer system—the role of patient experience. Aust J Cancer Nurs 13(2):4–8Google Scholar
  2. 2.
    Sizmur S, Redding D (2009) Core domains for measuring inpatients’ experience of care. Picker Institute Europe, OxfordGoogle Scholar
  3. 3.
    Gerteis M, Edgman-Levitan S, Daley J, Delbanco TL (1993) Through the patients eyes. Jossey-Bass, San FranciscoGoogle Scholar
  4. 4.
    Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C (2011) Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One 6(6):e20476CrossRefGoogle Scholar
  5. 5.
    Young JM, Walsh J, Butow PN, Solomon MJ, Shaw J (2011) Measuring cancer care coordination: development and validation of a questionnaire for patients. BMC Cancer 11:298CrossRefGoogle Scholar
  6. 6.
    Saunders C, Carter DJ, Bichel-Findlay J, Jordan A (2015) Cancer patient experience indicators—a review for the Cancer Institute NSW through the Sax Institute. Centre for Health Services Management. University of Technology Sydney, Sydney, Faculty of HealthGoogle Scholar
  7. 7.
    Saunders C, Carter DJ, Jordan A, Duffield C, Bichel-Findlay J (2016) Cancer patient experience measures: an evidence review. J Psychosoc Oncol 34(3):200–222CrossRefGoogle Scholar
  8. 8.
    NHS England (2015) CCG outcomes indicator set 2015/16: at-a-glance guide. Commissioning Strategy Directorate, NHS EnglandGoogle Scholar
  9. 9.
    Lacey K, Schofield PE, Krishnasamy M, Lethborg C, Cashill E, Thompson E, Butty J, Mollo A, Morrone A, Byrne J, Bishop JF (2014) Self-reported cancer patient experience in Australia and England. Asia-Pac J Clin Oncol 10:233Google Scholar
  10. 10.
    Canadian Partnership Against Cancer (2014) The 2014 cancer system performance report. Canadian Partnership Against Cancer, Toronto, OntarioGoogle Scholar
  11. 11.
    Ouwens M, Hermens R, Hulscher M, Vonk-Okhuijsen S, Tjan-Heijnen V, Termeer R, Marres H, Wollersheim H, Grol R (2010) Development of indicators for patient-centred cancer care. Support Care Cancer 18(1):121–130CrossRefGoogle Scholar
  12. 12.
    Uphoff EP, Wennekes L, Punt CJ, Grol RP, Wollersheim HC, Hermens RP, Ottevanger PB (2012) Development of generic quality indicators for patient-centered cancer care by using a RAND modified Delphi method. Cancer Nurs 35(1):29–37CrossRefGoogle Scholar
  13. 13.
    Chung KP, Lai MS, Cheng SH, Tang ST, Huang CC, Cheng AL, Hsieh PC (2008) Organization-based performance measures of cancer care quality: core measure development for breast cancer in Taiwan. Eur J Cancer Care 17(1):5–18Google Scholar
  14. 14.
    Miyashita M, Nakamura A, Morita T, Bito S (2008) Identification of quality indicators of end-of-life cancer care from medical chart review using a modified Delphi method in Japan. Am J Hosp Palliat Care 25(1):33–38CrossRefGoogle Scholar
  15. 15.
    Efstathiou N, Ameen J, Coll AM (2007) Healthcare providers’ priorities for cancer care: a Delphi study in Greece. Eur J Oncol Nurs 11(2):141–150CrossRefGoogle Scholar
  16. 16.
    Efstathiou N, Coll AM, Ameen J, Daly W (2011) Do Greek healthcare users and healthcare providers share cancer care priorities? Analysing the results from two Delphi studies. Eur J Cancer Care 20(2):179–186CrossRefGoogle Scholar
  17. 17.
    Sinha IP, Smyth RL, Williamson PR (2011) Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med 8(1):1–5CrossRefGoogle Scholar
  18. 18.
    von der Gracht HA (2012) Consensus measurement in Delphi studies: review and implications for future quality assurance. Technol Forecast Soc Chang 79(8):1525–1536CrossRefGoogle Scholar
  19. 19.
    White V, Johnson K (2011) Victorian patient experience survey development project: review of literature relating to survey tools used to assess cancer patients’ experiences of care. Prepared for the cancer strategy and development section. Department of Health, VictoriaGoogle Scholar
  20. 20.
    Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG (2009) Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 42(2):377–381CrossRefGoogle Scholar
  21. 21.
    Aldenderfer MS, Blashfield RK (1984) Cluster analysis. Sage University paper series on quantitative applications in the social sciences 07-044. Newbury Park, California, Sage PublicationsGoogle Scholar
  22. 22.
    Scottish Cancer Taskforce National Cancer Quality Steering Group (2013) Cancer patient experience quality performance indicators. NHS Scotland, Healthcare Improvement ScotlandGoogle Scholar
  23. 23.
    Keeney S, Hasson F, McKenna H (2006) Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs 53(2):205–212CrossRefGoogle Scholar
  24. 24.
    Sumsion T (1998) The Delphi technique: an adaptive research tool. Br J Occup Ther 61(4):153–156CrossRefGoogle Scholar
  25. 25.
    Hsu CC, Sandford BA (2007) The Delphi technique: making sense of consensus. Pract Assess Res Eval 12(10):1–8Google Scholar
  26. 26.
    Delnoij DM, Rademakers JJ, Groenewegen PP (201) The Dutch consumer quality index: an example of stakeholder involvement in indicator development. BMC Health Serv Res 10:88Google Scholar
  27. 27.
    Epstein RM, Street RL (2011) The values and value of patient-centered care. Ann Fam Med 9(2):100–103CrossRefGoogle Scholar
  28. 28.
    Sansoni J, Williams K, Morris D, Thompson C (2016) Development of prioritised patient experience indicators for the patient centred quality cancer system program: technical report. Centre for Health Service Development. University of Wollongong, Wollongong, Australian Health Services Research InstituteGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  • Kathryn E. Williams
    • 1
    Email author
  • Janet Sansoni
    • 1
  • Darcy Morris
    • 1
  • Cristina Thompson
    • 1
  1. 1.Centre for Health Service DevelopmentAustralian Health Services Research Institute (AHSRI), University of WollongongWollongongAustralia

Personalised recommendations