How Much Should We Trust Micro-data? A Comparison of the Socio-demographic Profile of Malawian Households Using Census, LSMS and DHS data

Original Article

Abstract

This paper assesses the empirical representativeness of micro-data by comparing the Malawi 2008 census to two representative household surveys – ‘the Living Standard Measurement Survey’ and the ‘Demographic and Health Survey’ – both implemented in Malawi in 2010. The comparison of descriptive statistics – demographics, asset ownership, and living conditions – shows considerable similarities despite statistically identifiable differences due to the large samples. Differences mainly occur when wording, scope, and pre-defined answer categories diverge across surveys. Multivariate analyses are considerably less representative due to loss of observations with composite indicators yielding higher comparability as individual ones. Household-level fixed-effect specifications produce more similar results, yet are not suited for policy conclusions. Comparability of micro-data should not be assumed but checked on a case-by-case basis. Still, micro-data constitute reliable grounds for factually informed conclusions if design and context are appropriately considered.

Keywords

household data survey representativeness sub-Saharan Africa Malawi 

Ce papier évalue la représentativité empirique des micro-données en comparant le recensement du Malawi de 2008 avec deux enquêtes représentatives des ménages – ‘l’Enquête de la Mesure des Niveaux de Vie’ (EMNV) et ‘l’Enquête sur la Démographie et la Santé’ (EDS) - qui ont chacune été mises en œuvre au Malawi en 2010. La comparaison des statistiques descriptives – la démographie, la propriété des biens et les conditions de vie - présente des similarités considérables malgré des différences statistiquement identifiables en raison des grands échantillons. Les différences se produisent principalement quand les formulations, la portée et les catégories de réponses prédéfinies divergent selon les enquêtes. Les analyses multivariées sont considérablement moins représentatives en raison de la perte d’observations ayant des indicateurs composites. Les spécifications à effets fixes au niveau des ménages produisent des résultats plus similaires, mais ne sont pas adaptées aux conclusions des politiques. La comparabilité des micro-données ne doit pas être présumée mais vérifiée au cas par cas. Néanmoins, les micro-données constituent des motifs fiables pour produire des conclusions factuelles si la conception et le contexte sont pris en considération.

References

  1. Akbulut-Yuksel, M. and Belgi, T. (2013) Left behind: Intergenerational transmission of human capital in the midst of HIV/AIDS. Journal of Population Economics 26(4): 1523–1547.CrossRefGoogle Scholar
  2. Alderman, H., Chiappori, P.A., Haddad, L., Hoddinott, J. and Kanbur, R. (1995) Unitary versus collective models of the household: Is it time to shift the burden of proof? World Bank Research Observer 10(1): 1–19.CrossRefGoogle Scholar
  3. Alwin, D.F. (1989) Problems in the estimation and interpretation of the reliability of survey data. Quality and Quantity 23: 277–331.CrossRefGoogle Scholar
  4. Bound, J., Brown, C. and Mathiowetz, N. (2001) Measurement error in survey data. In: J.J. Heckman and E. Leamer (eds.) Handbook of Econometrics Volume 5: 3705–3843. Amsterdam, The Netherlands: Elsevier.Google Scholar
  5. Brick, J.M. and Kalton, G. (1996) Handling missing data in survey research. Statistical Methods in Medical Research 5(3): 215–238.CrossRefGoogle Scholar
  6. Cameron, A.C. and Miller, D.L. (2015) A practitioner’s guide to cluster-robust inference. Journal of Human Resources 50(2): 317–372.CrossRefGoogle Scholar
  7. Checkley, W., Gilman, R.H., Black, R.E., Epstein, L.D., Cabrera, L., Sterling, C.R. and Moulton, L.H. (2004) Effect of water and sanitation on childhood health in a poor Peruvian peri-urban community. The Lancet 363(9403): 112–118.CrossRefGoogle Scholar
  8. Chen, Y. and Li, H. (2009) Mother’s education and child health: Is there a nurturing effect? Journal of Health Economics 28(2): 413–426.CrossRefGoogle Scholar
  9. Deaton, A. (1997) The Analysis of Household Surveys: A Microeconometric Approach to Development Policy. Washington, D.C: World Bank Publications.Google Scholar
  10. Deaton, A. (2005) Measuring poverty in a growing world (or measuring growth in a poor world. Review of Economics and Statistics 87(1): 1–19.CrossRefGoogle Scholar
  11. Desai, S. and Alva, S. (1998) Maternal education and child health: Is there a strong causal relationship? Demography 35(1): 71–81.CrossRefGoogle Scholar
  12. Duflo, E. (2003) Grandmothers and granddaughters: Old-age pensions and intrahousehold allocation in South Africa. World Bank Economic Review 17(1): 1–25.CrossRefGoogle Scholar
  13. Elbers, C., Lanjouw, J.O. and Lanjouw, P. (2003) Micro-level estimation of poverty and inequality. Econometrica 71(1): 355–364.CrossRefGoogle Scholar
  14. Epple, D., Romano, R.E. and Urquiola, M. (2015) School vouchers: A survey of the economics literature. National Bureau of Economic Research, Working Paper Series no. 21523.Google Scholar
  15. Fernald, L.C.H., Gertler, P. and Neufeld, L.M. (2009) 10-Year effect of Oportunidades, Mexico’s conditional cash transfer programme, on child growth, cognition, language, and behavior: A longitudinal follow-up study. The Lancet 374(9706): 1997–2005.CrossRefGoogle Scholar
  16. Fox, L. and Pimhidzai, O. (2013) Different Dreams, Same Bed: Collecting, Using, and Interpreting Employment Statistics in Sub-Saharan Africa—the Case of Uganda. World Bank Policy Research Working Paper No. 6436.Google Scholar
  17. Garg, A. and Morduch, J. (1998) Sibling rivalry and the gender gap: Evidence from child health outcomes in Ghana. Journal of Population Economics 11(4): 471–493.CrossRefGoogle Scholar
  18. Gertler, P. (2004) Do conditional cash transfers improve child health? Evidence from PROGRESA’s controlled randomized experiment. American Economic Review 94(2): 336–341.CrossRefGoogle Scholar
  19. Gondwe, T.N. and Wollny, C.B.A. (2007) Local chicken production system in Malawi: Household flock structure, dynamics, management and health. Tropical Livestock Health and Production 39(2): 103–113.CrossRefGoogle Scholar
  20. Gomes Victora, C., de Onis, M. Hallal, P.C., Blössner, M. and Shrimpton, R. (2010) Worldwide timing of growth faltering: Revisiting implications for interventions. Pediatrics 125(3): e473–e480.CrossRefGoogle Scholar
  21. Griliches, Z. (1986) Economic data issues. In: Z. Griliches and M. Intriligator (eds.) Handbook of Econometrics Volume 3: 1465–1514. Amsterdam, The Netherlands: Elsevier.Google Scholar
  22. Grosh, M.E. and Glewwe, P. (1996) Household survey data from developing countries: progress and prospects. The American Economic Review 86(2): 15–19.Google Scholar
  23. Guarcello, L., Kovrova, I., Lyon, S., Manacorda, M. and Rosati, F. C. (2010) Towards Consistency in Child Labour Measurement: Assessing the Comparability of Estimates Generated by Different Survey Instruments. Understanding Children’s Work Programme Working Paper.Google Scholar
  24. Heckman, J.T. (2001) Micro data, heterogeneity, and the evaluation of public policy: Nobel Lecture. Journal of Political Economy 109(4): 673–748.CrossRefGoogle Scholar
  25. Holt, D. (1985) Review of planning and analysis of observational studies by William E. Cochran. Journal of the American Statistical Association 80: 772–73.CrossRefGoogle Scholar
  26. Jerven, M. (2013) Poor Numbers: How We are Misled by African Development Statistics and What to do About it. Ithaca: Cornell University Press.Google Scholar
  27. Langkamp, D.L., Lehman, A. and Lemeshow, S. (2010) Techniques for handling missing data in secondary analyses of large surveys. Academic Pediatrics 10(3): 205–210.CrossRefGoogle Scholar
  28. Little, R.J.A. and Rubin, D.B. (2002) Statistical Analysis with Missing Data. Second edition, Hoboken, NJ: John Wiley & Sons.Google Scholar
  29. Manley, J., Gitter, S. and Slavchevska, V. (2013) How effective are cash transfers at improving nutritional status? World Development 48: 133–155.CrossRefGoogle Scholar
  30. McKenzie, D.J. (2005) Measuring inequality with asset indicators. Journal of Population Economics 18(2): 229–260.CrossRefGoogle Scholar
  31. Morgenstern, O. (1963) On the accuracy of economic observations. Second edition. Princeton, NJ: Princeton University Press.Google Scholar
  32. National Statistical Office (2008) Population and housing census (preliminary report). Zomba. Malawi.Google Scholar
  33. National Statistical Office (2012) Integrated household survey 2010–2011; Household socio-economic characteristics (report). Zomba. Malawi.Google Scholar
  34. Orcutt, G.H. (1962) Microanalytic models of the United States economy: Need and development. The American Economic Review 52(2): 229–240.Google Scholar
  35. Picard, N. and Wolff, F.C. (2010) Measuring educational inequalities: a method and an application to Albania. Journal of Population Economics 23(3): 989–1023.CrossRefGoogle Scholar
  36. Pongou, R., Ezzati, M. and Salomon, J.A. (2006) Household and community socioeconomic and environmental determinants of child nutritional status in Cameroon. BMC Public Health 6:98.CrossRefGoogle Scholar
  37. Randolph, T.F., Schelling, E., Grace, D., Nicholson, C.F., Leroy, J.L., Cole, D.C., Demment, M. W., Omore, A., Zinsstag, J. and Ruel, M. (2007) Role of livestock in human nutrition and health for poverty reduction in developing countries. Journal of Livestock Science 85(11): 2788–2800.Google Scholar
  38. Ravallion, M. and Chen, S. (1997) What can new survey data tell us about recent changes in distribution and poverty? World Bank Economic Review 11(2): 357–382.CrossRefGoogle Scholar
  39. Ravallion, M. (2003) The debate on globalization, poverty and inequality: why measurement matters. International Affairs 79(4): 739–753.CrossRefGoogle Scholar
  40. Rieger, M. and Wagner, N. (2015) Child health, its dynamic interaction with nutrition and health memory—Evidence from Senegal. Economics & Human Biology 16: 135–145.CrossRefGoogle Scholar
  41. Sandefur, J. and Glassman, A. (2015) The political economy of bad data: Evidence from African survey and administrative statistics. The Journal of Development Studies 51(2): 116–132.CrossRefGoogle Scholar
  42. Schoumaker, B. (2011) Omissions of births in DHS birth histories in sub-Saharan Africa: Measurement and determinants. Proceedings of the 2011 Annual Meeting of the Population Association of America; 31 March–2 April, Washington D.C.Google Scholar
  43. Schultz, T.P. (2002) Why governments should invest more to educate girls. World Development 13: 827–846.Google Scholar
  44. Smith, L.C., Ruel, M.T., and Ndiaye, A. (2005) Why is child malnutrition lower in urban than in rural areas? Evidence from 36 developing countries. World Development 33(8): 1285–1305.CrossRefGoogle Scholar
  45. Srinivasan, T.N. (1994) Data base for development analysis: An overview. Journal of Development Economics 44(1): 3–27.CrossRefGoogle Scholar
  46. Sutherland, H., Taylor, R. and Gomulka, J. (2002) Combining household income and expenditure data in policy simulations. Review of Income and Wealth 48(4): 517–536.CrossRefGoogle Scholar
  47. WHO (2010) World health statistics 2010. Geneva, Switzerland: World Health Organization.Google Scholar
  48. WHO and UNICEF (2009) WHO child growth standards and the identification of severe acute malnutrition in infants and children. A joint statement by the World Health Organization and the United Nations Children’s Fund. Geneva, Switzerland: WHO Press. Available at <http://www.who.int/nutrition/publications/severemalnutrition/9789241598163_eng.pdf>.
  49. World Bank (2015) World Development Indicators—GDP per capita (current US$). World Bank national accounts data. Washington, D.C.: World Bank.Google Scholar
  50. Yarnoff, B. (2011) Household allocation decisions and child health: can behavioral responses to vitamin A supplementation programmes explain heterogeneous effects? Journal of Population Economics 24(2): 657–680.CrossRefGoogle Scholar

Copyright information

© European Association of Development Research and Training Institutes (EADI) 2017

Authors and Affiliations

  1. 1.International Institute of Social Studies of Erasmus University RotterdamThe HagueThe Netherlands
  2. 2.School of Oriental and African Studies (SOAS)LondonUK

Personalised recommendations