Advertisement

Journal of Family and Economic Issues

, Volume 35, Issue 4, pp 516–531 | Cite as

Evaluation of a Financial Literacy Test Using Classical Test Theory and Item Response Theory

  • Irina A. Kunovskaya
  • Brenda J. Cude
  • Natalia Alexeev
Original Paper

Abstract

The purpose of this study was to evaluate the quality (item difficulty and discriminability, construct validity, and reliability) of a financial literacy test that has been used to measure financial literacy in multiple countries. The test was analyzed based on Classical Test Theory and Item Response Theory, specifically the Rasch model, with the goal of identifying areas of improvement. The findings from the two measurement frameworks were quite comparable and identified similar measurement problems. Steps to improve the financial literacy measure were recommended.

Keywords

Measurement of financial literacy Classical Test Theory The Rasch model 

References

  1. Allen, M., & Yen, W. (1979). Introduction to measurement theory. Monterey, CA: Brooks/Cole.Google Scholar
  2. Ardic, O. P., Ibrahim, J. A., & Mylenko, N. (2011). Consumer protection laws and regulations in deposit and loan services: A cross-country analysis with a new data set (Policy Research Working Paper 5536). The World Bank. doi: 10.1596/1813-9450-5536.
  3. Atkinson, A. (2008). Evidence of impact: An overview of financial education evaluations. University of Bristol: Personal Finance Research Centre. Retrieved January 12, 2012, from www.fsa.gov.uk/pubs/consumer-research/crpr68.pdf.
  4. Atkinson, A. (2012). OECD/INFE tools for cross-country surveys of financial literacy. Colombia-OECD-WB Conference on Financial Education. Cartagena, Colombia: OECD. Retrieved November 4, 2012, from http://www.oecd.org/daf/financialmarketsinsuranceandpensions/financialeducation/oecd-colombiainternationalconferenceonfinancialeducation.htm.
  5. Atkinson, A., McKay, S., Kempson, E., & Collard, S. (2006). Levels of financial capability in the UK: Results of a baseline survey. Financial Services Authority. Retrieved January 12, 2012, from www.fsa.gov.uk/pubs/consumer-research/crpr47.pdf.
  6. Atkinson, A., & Messy, F. (2011). Assessing financial literacy in 12 countries: An OECD pilot exercise. Journal of Pension Economics and Finance, 110, 657–665. doi: 10.1017/S1474747211000539.CrossRefGoogle Scholar
  7. Azerbaijan Micro-finance Association. (2010). Final report: Results of the financial literacy survey. Baku, Azerbaijan: Author. Retrieved November 20, 2011, from http://siteresources.worldbank.org/INTECAREGTOPPRVSECDEV/Resources/Financial_Literacy_Survey_Report_EN.pdf.
  8. Baker, F. (2001). The basics of item response theory. ERIC clearinghouse on assessment and evaluation. College Park: University of Maryland.Google Scholar
  9. Bond, T. G., & Fox, C. M. (2001). Applying the Rasch model: Fundamental measurement in the human sciences. London: Lawrence Erlbaum Associates.Google Scholar
  10. Borden, L. M., Lee, S., Serido, J., & Collins, D. (2008). Changing college students’ financial knowledge attitudes and behavior through seminar participation. Journal of Family and Economic Issues, 29, 23–40. doi: 10.1007/s10834-007-9087-2.CrossRefGoogle Scholar
  11. Carpena, F., Cole, S., Shapiro, J., & Zia, B. (2012). Unpacking the causal chain of financial literacy (Policy Research Working Paper 5798). Washington, DC: The World Bank.Google Scholar
  12. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  13. Cole, S., Sampson, T., & Zia, B. (2009). Financial literacy, financial decisions, and the demand for financial services: Evidence from India and Indonesia (Working Paper 09-117). Harvard Business School, Harvard University. Retrieved January 12, 2012, from http://ssrn.com/abstract=1374078.
  14. Cronbach, L. J. (1958). Proposals leading to analytic treatment of social perception scores. In R. Tagiuri & L. Petrullo (Eds.), Person perception and interpersonal behavior (pp. 353–379). Stanford, CA: Stanford University Press.Google Scholar
  15. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  16. Fan, X. (1998). Item response theory and classical test theory: An empirical comparison of their item/person statistics. Educational and Psychological Measurement, 58(3), 357–381. doi: 10.1177/0013164498058003001.CrossRefGoogle Scholar
  17. Green, K., & Frantom, C. (2002, November). Survey development and validity with Rasch model. A paper presented at the International Conference on Questionnaire Development, Evaluation, and Testing. Charleston, SC. Retrieved January 12, 2012, from http://www.jpsm.umd.edu/qdet/final_pdf_papers/green.pdf.
  18. Hagquist, C., Bruce, M., & Gustavson, P. (2009). Using the Rasch model in nursing research: An introduction and illustrative example. Journal of Nursing Studies, 46, 380–393. doi: 10.1016/j.ijnurstu.2008.10.007.CrossRefGoogle Scholar
  19. Hambleton, R. K. (2000). Emergence of item response modeling in instrument development and data analysis. Medical Care, 38(Suppl II), II-60–II-65. doi: 10.1097/00005650-200009002-00009.Google Scholar
  20. Hambleton, R. K. (2005). Issues, designs and technical guidelines for adapting tests into multiple languages and cultures. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberg (Eds.), Adapting educational and psychological tests to cross-cultural assessment (pp. 3–38). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  21. Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff Publishing.CrossRefGoogle Scholar
  22. Hardouin, J.-B. (2007). Rasch analysis: Estimation and tests with Rasch test. Stata Journal, 7(1), 22–44.Google Scholar
  23. Hastings, J. S., Madrian, B. C., & Skimmyhorn, W. L. (2012). Financial literacy, financial education and economic outcomes (NBER Working Paper 18412). Retrieved December 4, 2011, from http://www.nber.org/papers/w18412.
  24. Holzman, R. (2010). Bringing financial literacy education to low and middle income countries. The need to review, adjust, and extend current wisdom (Discussion Paper No. 5114). Bonn: The Institute for the Study of Labor (IZA). Retrieved December 4, 2011, from http://ssrn.com/abstract=1663134.
  25. Hung, A. A., Parker, A. M., & Yoong, J. K. (2009). Defining and measuring financial literacy (RAND Corporation Working Paper 708). Retrieved December 4, 2011, from http://www.rand.org/pubs/working_papers.
  26. Huston, S. (2010). Measuring financial literacy: A proposed approach. Journal of Consumer Affairs, 45(2), 296–316. doi: 10.1111/j.1745-6606.2010.01170.x.CrossRefGoogle Scholar
  27. Jump$tart Coalition. (2007). National standards in K-12 personal finance education. Retrieved December 4, 2011, from http://www.jumpstart.org/national-standards.html.
  28. Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141–151. doi: 10.1177/001316446002000116.CrossRefGoogle Scholar
  29. Kempson, E. (2009). Framework for the development of financial literacy baseline surveys: A first international comparative analysis. Paris, France: OECD. Retrieved November 2, 2012, from www.oecd.org/daf/fin/wp.
  30. Kempson, E., Scott, V., & Perotti, V. (2012). Measuring financial capability in a low- and middle-income setting. Colombia-OECD-WB Conference on Financial Education. Cartagena, Colombia: OECD. Retrieved November 2, 2012, from http://www.oecd.org/daf/financialmarketsinsuranceandpensions/financialeducation/oecd-colombiainternationalconferenceonfinancialeducation.htm.
  31. Kim, J.-O., & Mueller, C. W. (1978). Factor analysis: Statistical methods and practical issues. Newbury Park: SAGE.Google Scholar
  32. Kimberlin, C., & Winterstein, A. (2008). Validity and reliability of measurement instruments used in research. American Journal of Health-System Pharmacy, 65(23), 2276–2284.Google Scholar
  33. Klapper, L. F., Lusardi, A., & Panos, G. A. (2011). Financial literacy and the financial crisis: Evidence from Russia. Retrieved December 4, 2011, from http://ssrn.com/abstract=1786826 or http://dx.doi.org/10.2139/ssrn.1786826.
  34. Knoll, M., & Houts, C. (2012). The financial knowledge scale: An application of Item Response Theory to the assessment of financial literacy. Journal of Consumer Affairs, 46(3), 381–410.CrossRefGoogle Scholar
  35. Kolenikov, S., & Angeles, G. (2004). The use of discrete data in PCA: Theory, simulations, and applications to socioeconomic indices (Working Paper 04-85). MEASURE/Evaluation Project. University of North Carolina, Chapel Hill: Carolina Population Center. Retrieved December 4, 2011, from http://www.cpc.unc.edu/measure/publications/wp-04-85.
  36. Kunovskaya, I., & Cude, B. (2012, April). Comparative analysis of money management as a component of financial capability in transitional economies. In Proceedings of the 2012 American council on consumer interests annual conference. Retrieved January 4, 2013, from http://www.consumerinterests.org/pdffiles/2012-conference/2012-9%20Comparative%20Analysis%20of%20Money%20Management.pdf.
  37. Linacre, J. M. (2009). A user’s guide to WINSTEPS: MINISTEPS Rasch Model computer programs (Program Manual 3.69.0). Retrieved from http://www.winsteps.com/index.htm.
  38. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.Google Scholar
  39. Lucey, T. (2005). Assessing the reliability and validity of the Jump$tart survey of financial literacy. Journal of Family and Economic Issues, 26(2), 283–284. doi: 10.1007/s10834-005-3526-8.CrossRefGoogle Scholar
  40. Lusardi, A., & Mitchell, O. S. (2005). Financial literacy and planning: Implications for retirement wellbeing (Michigan Retirement Research Center Research Paper No. WP 2005-108). doi:  10.2139/ssrn.881847.
  41. Lusardi, A., & Mitchell, O. S. (2007). Baby boomer retirement security: The roles of planning, financial literacy, and housing wealth. Journal of Monetary Economics, 54(1), 205–224. doi: 10.1016/j.jmoneco.2006.12.001.CrossRefGoogle Scholar
  42. Lusardi, A., & Mitchell, O. S. (2011). Financial literacy around the world: Introduction and overview (CeRP Working Paper 106/11). Retrieved January 20, 2012, from http://www.cerp.carloalberto.org/en/publications.
  43. Lusardi, A., & Tufano, P. (2009). Debt literacy, financial experiences, and overindebtedness (NBER Working Paper 14808). Retrieved February 10, 2011, from http://www.nber.org/papers.
  44. Macayan, J., & Ofalia, B. (2011). Determining experts and novices in college algebra: A psychometric test development and analysis using the Rasch model (1PL-IRT). Educational Measurement and Evaluation Review, 2, 62–76.Google Scholar
  45. Magno, C. (2009). Demonstrating the difference between classical test theory and item response theory using derived test data. International Journal of Educational and Psychological Assessment, 1(1), 1–11. Retrieved December 4, 2011, from http://ssrn.com/abstract=1426043.
  46. Mandell, L. (2008). Financial literacy of high school students. In J. J. Xiao (Ed.), Handbook of consumer finance research (pp. 163–183). New York, NY: Springer. doi: 10.1007/978-0-387-75734-6_10.CrossRefGoogle Scholar
  47. Moore, D. (2003). Survey of financial literacy in Washington State: Knowledge, behavior, attitudes, and experiences (Technical Report n. 03-39). Washington State University, Social and Economic Sciences Research Center. Retrieved February 14, 2009, from www.dfi.wa.gov/news/finlitsurvey.pdf.
  48. Noctor, M., Stoney, S., & Stradling, S. (1992). Financial literacy: A discussion of concepts and competences of financial literacy and opportunities for its introduction into young people’s learning (Report prepared for the National Westminster Bank). London: National Foundation for Education Research.Google Scholar
  49. Organisation for Economic Co-operation and Development (OECD). (2005). Improving financial literacy: Analysis of issues and policies. Paris, France: OECD. Retrieved January 20, 2012, from http://www.oecd.org/document/2/0,3746,en_2649_15251491_35802524_1_1_1_1,00.html.
  50. Organisation for Economic Co-operation and Development (OECD). (2009). PISA 2009 results: What students know and can do. Paris, France: OECD. Retrieved November 5, 2012, from www.oecd.org/edu/pisa/2009.
  51. Organisation for Economic Co-operation and Development (OECD). (2012). PISA 2012 financial literacy framework. Paris, France: OECD. Retrieved November 5, 2012, from www.oecd.org/dataoecd/8/43/46962580.pdf.
  52. Pellinen, A., Törmäkangas, K., Uusitalo, O., & Raijas, A. (2011). Measuring the financial capability of investors: A case of the customers of mutual funds in Finland. International Journal of Bank Marketing, 29(2), 107–133. doi: 10.1108/02652321111107611.CrossRefGoogle Scholar
  53. Perotti, V. (2012, October). Review of existing financial capability and literacy measurement instruments. Colombia-OECD-WB Conference on Financial Education. Cartagena, Colombia: OECD. Retrieved November 5, 2012, from http://www.oecd.org/daf/financialmarketsinsuranceandpensions/financialeducation/oecd-colombiainternationalconferenceonfinancialeducation.htm.
  54. Perotti, V., Siegfried, Z., Larassi, G., & Bolaji-Adio, A. (2013). Making sense of financial capability surveys around the world. Washington: International Bank for Reconstruction and Development. Retrieved June 12, 2013, from http://responsiblefinance.worldbank.org/~/media/GIAWB/FL/Documents/Misc/Financial-Capability-Review.pdf.
  55. Postmus, J., Plummer, S., McMahon, S., & Zurlo, K. (2012). Financial literacy: Building economic empowerment with survivors of violence. Journal of Family and Economic Issues, 34, 275–284. doi: 10.1007/s10834-012-9330-3.CrossRefGoogle Scholar
  56. President’s Advisory Council on Financial Literacy. (2008). 2008 Annual report to the President. Washington, DC: The Department of the Treasury. Retrieved January 20, 2012, from http://www.treasury.gov/resource-center/financial-education/Documents/PACFL_ANNUAL_REPORT_1-16-09.pdf.
  57. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danish Institute for Educational Research.Google Scholar
  58. Rasch, G. (1993). Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago.Google Scholar
  59. Remund, D. L. (2010). Financial literacy explicated: The case for a clearer definition in an increasingly complex economy. Journal of Consumer Affairs, 44(2), 276–295. doi: 10.1111/j.1745-6606.2010.01169.x.CrossRefGoogle Scholar
  60. Social and Enterprise Development Innovations (SEDI). (2004, July). Financial capability and poverty: New approaches for addressing poverty and exclusion (Discussion Paper). Toronto, Canada: Author. Retrieved January 20, 2012, from http://www.sedi.org/DataRegV2-unified/sedi-Publications/PRIEnglishAug2004.pdf.
  61. Stănculescu, M. (2010, June). Analysis of the financial literacy survey in Romania and recommendations. Bucharest, Romania: Institute for the Study of the Quality of Life. Retrieved January 20, 2012, from http://siteresources.worldbank.org/INTECAREGTOPPRVSECDEV/Resources/Romania_Financial_Literacy_June_2010.pdf.
  62. van Rooij, M., Lusardi, A., & Alessie, R. (2011). Financial literacy and stock market participation. Journal of Financial Economics, 101(2), 449–472. doi: 10.1016/j.jfineco.2011.03.006.CrossRefGoogle Scholar
  63. Varma, S. (2010). Preliminary item statistics using point-biserial correlation and p-values. Educational Data Systems, Inc. Retrieved January 20, 2012, from http://www.eddata.com/resources/publications.
  64. Washburn, I. J. (2009). Rasch modeling in family studies: Modification of the relationship assessment scale. (Master’s thesis) Retrieved January 20, 2012, from http://ir.library.oregonstate.edu/xmlui/bitstream/handle/1957/11182/Take%20the%20Relationship%20Assessment%20Scale%20and%20redesign%20it%20using%20Rasch%20Modeling.pdf?sequence=1.
  65. Wiberg, M. (2004). Classical test theory vs. item response theory: An evaluation of the theory test in the Swedish driving license-test. Umea: Umea Universiteit. Retrieved January 20, 2012, from http://www.jus.umu.se/digitalAssets/59/59529_em-no-50.pdf.
  66. Wong, H. M., McGrath, C. P. J., & King, N. M. (2011). Rasch validation of the early childhood oral health impact scale. Community Dentistry and Oral Epidemiology, 39, 449–557. doi: 10.1111/j.1600-0528.2011.00614.x.CrossRefGoogle Scholar
  67. World Bank Group, Financial and Private Sector Development Vice Presidency, FPDFS—Financial Systems Policy Unit. (2011). Good practices for financial consumer protection (Consultative Draft). Washington, DC: Author. Retrieved January 20, 2012, from http://siteresources.worldbank.org/EXTFINANCIALSECTOR/Resources/Good_Practices_Financial_CP.pdf.
  68. Wright, B. D. (1994). Unidimensionality coefficient. Rasch Measurement Transactions, 8(3), 385.Google Scholar
  69. Wright, B. D. (1999). Fundamental measurement for psychology. In S. E. Embretson & S. L. Hershberger (Eds.), The new rules of measurement: What every psychologist and educator should know (pp. 65–104). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  70. Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago, IL: MESA Press.Google Scholar
  71. Yoong, J., Mihaly, K., Bauhoff, S., & Rabinovich, L. (2013). A toolkit for the evaluation of financial capability programs in low- and middle income countries. Washington: International Bank for Reconstruction and Development. Retrieved June 12, 2013, from http://documents.worldbank.org/curated/en/2013/01/18054632/toolkit-evaluation-financial-capability-programs-low-middle-income-countries.

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Irina A. Kunovskaya
    • 1
  • Brenda J. Cude
    • 2
  • Natalia Alexeev
    • 3
  1. 1.College of Family and Consumer SciencesUniversity of GeorgiaAthensUSA
  2. 2.College of Family and Consumer SciencesUniversity of GeorgiaAthensUSA
  3. 3.Department of MathematicsUniversity of GeorgiaAthensUSA

Personalised recommendations