Abstract
Surveillance system indicators of morbidity, mortality, or behaviors are used to provide regional information for ever-smaller areas, most recently for ranking counties. These rankings are thought to provide information about the relative standing of regions and provide information about problem areas and the success of programs. We investigate the ability of such rankings to reliably assess health. We assess the reliability of several ranked health indices used at the county level and the consistency of an index's quality across different states. Reliability is assessed using an index of reliability, simulations, and the ability of an index to consistently identify the top 10 % (worst) counties within a state. There is marked variability across measures to provide consistent ranks, across states, and, many times, across states for a particular measure. A few health measures do consistently well, e.g., Teen Birth, Chlamydia, and Years of Potential Life Lost rates. While a few health rankings appear worthy of use for policy and in the identification of local problems, in general, they are not consistent across regions.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Arndt, S., Acion, L., Caspers, K., & Diallo, O. (2011). Assessing community variation and randomness in public health indicators. Population Health Metrics, 9, 3. doi:10.1186/1478-7954-9-3.
Buehler, J. W., & Holtgrave, D. R. (2007). Challenges in defining an optimal approach to formula-based allocations of public health funds in the United States. BMC Public Health, 7, 44. doi:10.1186/1471-2458-7-44.
Cochran, W. G. (1954). The combination of estimates from different experiments. Biometrics, 10, 101–129.
Cochran, W. G. (1983). Planning and analysis of observational studies. New York: John Wiley & Sons.
DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188.
Evans, M., Hastings, N., & Peacock, B. (1993). Statistical distributions (2nd ed.). New York: John Wiley & Sons.
Flowers, J., Hall, P., & Pencheon, D. (2005). Mini-symposium—public health observatories: Public health indicators. Public Health, 119, 239–245.
Guilford, J. P. (1954). Psychometric methods. New York: McGraw-Hill.
Hall, P., & Miller, H. (2010). Modeling the variability of ranks. The Annals of Statistics, 38, 2652–2677.
Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539–1558. doi:10.1002/sim.1186.
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557–560. doi:10.1136/bmj.327.7414.557.
Hofer, T. P., Hayward, R. A., Greenfield, S., Wagner, E. H., Kaplan, S. H., & Manning, W. G. (1999). The unreliability of individual physician “report cards” for assessing the costs and quality of care of a chronic disease. JAMA: The Journal of the American Medical Association, 281, 2098–2105. doi:10.1001/jama.281.22.2098.
Hofer, T. P., & Haywood, R. A. (1996). Identifying poor-quality hospitals: Can hospital mortality rates detect quality problems for medical diagnoses? Medical Care, 34, 737–753.
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006). Assessing heterogeneity in meta-analysis: Q statistic or I² index? Psychological Methods, 11, 193–206. doi:10.1037/1082-989x.11.2.193.
Jacobs, R., Goddard, M., & Smith, P. C. (2005). How robust are hospital ranks based on composite performance measures? Medical Care, 43, 1177–1184.
Kephart, G., & Asada, Y. (2009). Need-based resource allocation: Different need indicators, different results? BMC Health Services Research, 9, 122. doi:10.1186/1472-6963-9-122.
Krieger, N., Chen, J. T., Waterman, P. D., Soobader, M.-J., Subramanian, S. V., & Carson, R. (2002). Geocoding and monitoring of us socioeconomic inequalities in mortality and cancer incidence: Does the choice of area-based measure and geographic level matter? American Journal of Epidemiology, 156, 471–482. doi:10.1093/aje/kwf068.
Krieger, N., Chen, J. T., Waterman, P. D., Soobader, M.-J., Subramanian, S. V., & Carson, R. (2003). Choosing area based socioeconomic measures to monitor social inequalities in low birth weight and childhood lead poisoning: The public health disparities geocoding project (US). Journal of Epidemiology and Community Health, 57, 186–199. doi:10.1093/aje/kwf068.
Louis, T. A., Jabine, T. B., & Gerstein, M. A. (2003). Statistical issues in allocating funds by formula. Washington, D.C.: National Academies Press.
Marshall, E. C., Sanderson, C., Spiegelhalter, D. J., & McKee, M. (1998). Reliability of league tables of in vitro fertilisation clinics: retrospective analysis of live birth rates Commentary: How robust are rankings? The implications of confidence intervals. British Medical Journal, 316, 1701–1705. doi:10.1136/bmj.316.7146.1701.
McGinnis, J.M. (2010). Observations on incentives to improve population health. Preventing Chronic Disease, 9, A92. Retreived from http://www.cdc.gov/pcd/issues/2010/sep/10_0078.htm.
Mokdad, A. H., & Remington, P. L. (2010). Measuring health behaviors in populations. Preventing Chronic Disease, 7, A75.
Muhuri, P.K., & Ducrest, J.L. (2007). Block grants and formula grants: A guide for allotment calculations. Rockville, MD: SAMHSA, Office of Applied Studies Retrieved from http://oas.samhsa.gov/BG_documentation_070809_final_psg.pdf.
Nunnally, J. C. (1967). Psychometric theory. New York: McGraw-Hill.
O'Brien, S. M., & Peterson, E. D. (2007). Identifying high-quality hospitals: Consult the ratings or flip a coin? Archives of Internal Medicine, 167, 1342–1344. doi:10.1001/archinte.167.13.1342.
Oliver, T.R. (2010). Population health rankings as policy indicators and performance measures. Preventing Chronic Disease, 5, A101. Retrieved from http://www.cdc.gov/pcd/issues/2010/sep/10_0040.htm.
Page, S., & Cramer, K. (2001). Maclean’s rankings of health care indices in canadian communities, 2000: Comparisons and statistical contrivance. Canadian Journal of Public Health, 92, 295–298.
Peppard, P. E., Kindig, D. A., Dranger, E., Jovaag, A., & Remington, P. L. (2008). Ranking community health status to stimulate discussion of local public health issues: The Wisconsin county health rankings. American Journal of Public Health, 98, 209–212. doi:10.2105/AJPH.2006.092981.
Remington, P. L., & Booske, B. C. (2011). Measuring the health of communities—how and why? Journal of Public Health Management and Practice, 17, 397–400. doi:10.1097/PHH.0b013e318222b897.
Rohan, A. M. K., Booske, B. C., & Remington, P. L. (2009). Using the Wisconsin county health rankings to catalyze community health improvement. Journal of Public Health Management and Practice, 15, 24–32. doi:10.1097/PHH.1090b1013e3181903bf3181908.
Vila, P., Booske, B., & Remington, P. (2006). Measuring mortality in the Wisconsin county health. Population Health Institute Technical Report., (Department of Population Health Sciences). Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.169.3689&rep=rep1&type=pdf
Funding
This project was funded, in part, by the Iowa Consortium for Substance Abuse Research and Evaluation.
Conflict of Interest
None of the authors have any financial disclosures.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Arndt, S., Acion, L., Caspers, K. et al. How Reliable Are County and Regional Health Rankings?. Prev Sci 14, 497–502 (2013). https://doi.org/10.1007/s11121-012-0320-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-012-0320-3