Environmental and Resource Economics

, Volume 72, Issue 2, pp 559–582 | Cite as

Reliability and Validity in Nonmarket Valuation

  • Richard C. Bishop
  • Kevin J. BoyleEmail author


We propose a framework for assessing the accuracy of nonmarket values. This involves adapting two widely-used concepts. Reliability addresses variance and validity addresses potential biases. These concepts are formally defined and adapted to assess the accuracy of individual nonmarket valuation studies and the potential accuracy of valuation methods. We illustrate the framework by considering, in a preliminary way, the reliability and validity of the contingent-valuation and travel-cost methods.


Nonmarket valuation Reliability Validity Contingent valuation Travel-cost method 



This article used Chapter 12 in Champ et al (2017) as a starting point. We thank Springer, the book’s publisher, for permission to do so. We also thank Ian Bateman for encouraging us to adapt and further develop the ideas in the chapter. Two anonymous reviewers provided many helpful comments and suggestions. Coral Bishop provided invaluable help in preparing the manuscript.


  1. Arrow K, Solow R, Portnoy PR, Leamer EE, Radner R, Schuman H (1993) Report of the NOAA panel on contingent valuation. Fed Reg 58:4601–4614Google Scholar
  2. Bhattacharjee S, Kling CL, Herriges JA (2009). Kuhn-Tucker estimation of recreation demand—a study of temporal stability. Agricultural and Applied Economics Association annual meetingGoogle Scholar
  3. Berrens RP (2000) Reluctant respondents and contingent valuation surveys. Appl Econ Lett 7:263–266Google Scholar
  4. Bockstael NE, Kling CL (1988) Valuing environmental quality: weak complementarity with sets of goods. Am J Agric Econ 70(3):654–662Google Scholar
  5. Bockstael NE, Hanemann WM, Kling CL (1987) Modelling recreational demand in a multiple site framework. Water Resour Res 23(5):951–960Google Scholar
  6. Boxall PC, Adamowicz WL (2002) Understanding heterogeneous preferences in random utility models: a latent class approach. Environ Resour Econ 23(4):421–446Google Scholar
  7. Boxall PC, Rollins K, Englin J (2003) Heterogeneous preferences for congestion during a wilderness experience. Resour Energy Econ 25:177–195Google Scholar
  8. Brander LM, Van Beukering P, Cesar HSJ (2007) The recreational value of coral reefs: a meta-analysis. Ecol Econ 63(1):209–218Google Scholar
  9. Brower R, Bateman IJ (2005) Temporal stability and transferability of willingness to pay for flood control and wetland conservation. Water Resour Res 46(2):353–361Google Scholar
  10. Brown WG, Nawas F (1973) Impact of aggregation on the estimation of outdoor recreation demand functions. Am J Agric Econ 55(2):246–249Google Scholar
  11. Carmines EG, Zeller RA (1979) Reliability and validity assessment. In: Sage university paper series on quantitative applications in the social sciences. Sage, Woburn, MAGoogle Scholar
  12. Carson R (2011) Contingent valuation: a comprehensive bibliography and history. Edward Elgar, CheltenhamGoogle Scholar
  13. Carson R, Flores NE, Martin K, Wright JL (1996) Contingent valuation and revealed preference methodologies: comparing the estimates for quasi-public goods. Land Econ 72:80–99Google Scholar
  14. Carson R, Flores NE, Meade NF (2001) Contingent valuation: controversies and evidence. Environ Resour Econ 19:173–210Google Scholar
  15. Carson R, Groves RM (2007) Incentive and informational properties of preference questions. Environ Resour Econ 37:181–210Google Scholar
  16. Carson R, Groves T, List JA (2014) Consequentiality: a theoretical and experimental exploration of a single binary choice. J Assoc Environ Resour Econ 1:171–207Google Scholar
  17. Carson R, Hanemann M, Kopp RJ, Krosnick JA, Mitchell RC, Presser S, Rudd PA, Smith VK, Conaway M, Martin K (1997) Temporal reliability of estimates from contingent valuation. Land Econ 73(2):151–163Google Scholar
  18. Carson R, Mitchell RC, Hanemann M, Kopp RJ, Presser S, Ruud PA (2003) Contingent valuation and lost passive use: damages from the Exxon Valdez oil spill. Environ Resour Econ 1:171–207Google Scholar
  19. Cesario FJ, Knetsch JL (1970) Time bias in recreation benefit estimates. Water Resour Res 6(3):700–704Google Scholar
  20. Champ PA, Brown TC (1997) A comparison of contingent and actual voting behavior. In: Proceedings from W-133 benefits and cost transfer in natural resource planning, 10th Interim Report, pp 77–98Google Scholar
  21. Champ PA, Boyle KJ, Brown TC (eds) (2017) A primer on nonmarket valuation, Springer, New YorkGoogle Scholar
  22. Chu A, Eisenhower D, Hay M, Morganstein D, Neter J, Waksberg J (1992) Measuring the recall error in self-reported fishing and hunting activities. J Off Stat 8(1):19–39Google Scholar
  23. Desvousges WH, Matthews K, Train K (2012) Adequate responsiveness to scope in contingent valuation. Ecol Econ 84:121–128Google Scholar
  24. Diamond P (1996) Testing the internal consistency of contingent valuation surveys. J Environ Econ Manag 30(3):337–347Google Scholar
  25. Diamond PA, Hausman JA (1994) Contingent valuation: Is some number better than no number? J Econ Perspect 8:45–64Google Scholar
  26. Egan KJ, Herriges JA, Kling CL, Downing JA (2009) Valuing water quality as a function of water quality measures. Am J Agric Econ 91(1):106–123Google Scholar
  27. Gems B, Ghosh D, Hitlin R (1982) A recall experiment: impact of time on recall on recreational fishing trips. In: Proceedings of the section on survey research methods. American Statistical Association, Washington, DCGoogle Scholar
  28. Gen S (2004) Meta-analysis of environmental valuation studies. Public Policy, Georgia Institute of Technology, AtlantaGoogle Scholar
  29. Grijalva TR, Berrens RP, Bohara A, Shaw W (2002) Testing the validity of contingent behavior trip responses. Am J Agric Econ 84(2):401–414Google Scholar
  30. Haab TC, McConnell KE (2002) Valuing environmental and natural resources: the econometrics of non-market valuation. Edward Elgar, CheltenhamGoogle Scholar
  31. Haener MK, Boxall PC, Adamowicz WL, Kuhnke DH (2004) Aggregation bias in recreation site choice models: resolving the resolution problem. Land Econ 80(4):561–574Google Scholar
  32. Hanley N (1989) Valuing non-market goods using contingent valuation. J Econ Surv 3(3):235–252Google Scholar
  33. Hanley N, Schlapfer F, Spurgeon J (2003) Aggregating the benefits of environmental improvements: distance-decay functions for use and non-use values. J Environ Manag 68:297–304Google Scholar
  34. Hausman JA (1993) Contingent valuation: a critical assessment. Emerald Group Publishing Limited, BingleyGoogle Scholar
  35. Hausman JA (2012) Contingent valuation: from dubious to hopeless. J Econ Perspect 26:43–56Google Scholar
  36. Herriges JA, Kling CL (2008) Revealed preference approaches to environmental valuation, volumes I and II. Routledge, New YorkGoogle Scholar
  37. Holmes T, Adamowicz W, Carlsson F (2017) Choice experiments. In: Champ PA, Boyle KJ, Brown TC (eds) A primer on nonmarket valuation, Ch. 5. Springer, New YorkGoogle Scholar
  38. Hynes S, Hanley N, Garvey E (2007) Up the proverbial creek without a paddle: accounting for variable participant skill levels in recreational demand modelling. Environ Resour Econ 36(4):413–426Google Scholar
  39. Jacobsen JB, Hanley N (2009) Are there income effects on global willingness to pay for biodiversity conservation? Env Resour Econ 43(2):137–160Google Scholar
  40. Joen Y, Herriges JA, Kling CL, Downing JA (2011) The role of water quality perceptions in modelling lake recreation demand. In: Bennett JW (ed) The international handbook on non-market environmental valuation. Edward Elgar, Cheltenham, pp 74–101Google Scholar
  41. Johnston RJ (2006) Is hypothetical bias universal? Validating contingent valuation response using a binding public referendum. J Environ Econ Manag 52:469–481Google Scholar
  42. Johnston R, Boyle KJ, Adamowicz W, Bennett J, Brouwer R, Cameron TA, Hanemann WM, Hanley N, Ryan M, Scarpa R, Tourangeau R, Vossler CA (2017) Contemporary guidance for stated preference studies. J Assoc Environ Resour Econ 4(2):319–405Google Scholar
  43. Jones-Lee MW, Hammerton M, Philips PR (1985) The value of safety: results of a national sample survey. Econ J 95:49–72Google Scholar
  44. Kaoru Y (1995) Measuring marine recreation benefits of water quality improvements by the nested random utility model. Resour Energy Econ 17(2):119–136Google Scholar
  45. Kealy MJ, Dovidio JF, Rockel ML (1988) Accuracy in valuation is a matter of degree. Land Econ 64(2):158–171Google Scholar
  46. Kealy MJ, Montgomery M, Dovidio JF (1990) Reliability and predictive validity of contingent valuation: Does the nature of the good matter? J Environ Econ Manag 19:244–263Google Scholar
  47. Kim Y, Kling CL, Zhao J (2015) Understanding behavioral explanations for the WTP-WTA divergence through a neoclassical lens: implications for environmental policy. Ann Rev Resour Econ 7(1):169–187Google Scholar
  48. Kinnel JC, Bingham MF, Mohamed AF, Desvousges WH, Kiler TB, Hastings EK, Kuhns KT (2006) Estimating site choice decisions for urban recreators. Land Econ 82(2):257–272Google Scholar
  49. Landry CE, Hindsley P (2011) Valuing beach quality with hedonic property models. Land Econ 87(1):92–108Google Scholar
  50. Landry CE, Liu H (2009) A semi-parametric estimator for revealed and stated preference data—an application to recreational beach visitation. J Environ Econ Manag 57(2):205–218Google Scholar
  51. Lindhjem H, Navrud S (2009) Asking for individual or household willingness to pay for environmental goods? Environ Resour Econ 43(1):11–29Google Scholar
  52. List JA, Gallet CA (2001) What experimental protocol influence disparities between actual and hypothetical stated values? Evidence from a meta-analysis. Environ Resour Econ 20:241–254Google Scholar
  53. Little J, Berrens RP (2004) Explaining disparities between actual and hypothetical stated values: further investigation using meta-analysis. Econ Bull 3:1–13Google Scholar
  54. List JA, Gallet CA (2001) What experimental protocol influence disparities between actual and hypothetical stated values? Environ Resour Econ 20(3):241–254Google Scholar
  55. Loehman E, De VH (1982) Application of stochastic choice modeling to policy analysis of public goods: a case study of air quality improvements. Rev Econ Stat 54:474–480Google Scholar
  56. Loomis JB (1989) Test–retest reliability of the contingent valuation method: a comparison of general population and visitor responses. Am J Agric Econ 71:76–84Google Scholar
  57. Loomis JB (1990) Comparative reliability of the dichotomous choice and open-ended contingent valuation techniques. J Environ Econ Manag 18(1):78–85Google Scholar
  58. Loureiro ML, Loomis JB (2017) How sensitive are environmental valuations to economic downturns? Ecol Econ 140:235–240Google Scholar
  59. Lupi F, Hoehn JP, Christie GC (2003) Using an economic model of recreational fishing to evaluate the benefits of sea lamprey (Petromyzon marinus) control on the St Marys River. J Great Lakes Res 29:742–754Google Scholar
  60. Massey DM, Newbold SC, Gentner B (2006) Valuing water quality changes using a bioeconomic model of a coastal recreational fishery. J Environ Econ Manag 52(1):482–500Google Scholar
  61. Mazurkiewicz SM, Boyle KJ, Teisl MF, Morris KI, Clark AG (1996) Recall bias and reliability of survey data: moose hunting in Maine. Wildl Soc Bull 24(1):140–148Google Scholar
  62. McCollum DW (1986) The travel cost method: time, specification and validity. University of Wisconsin-Madison, MadisonGoogle Scholar
  63. McConnell KE (1977) Congestion and willingness to pay: a study of beach use. Land Econ 53(2):185–195Google Scholar
  64. McConnell KE, Strand IE, Valdes S (1998) Testing temporal reliability and carry-over effect: the role of correlated responses in test–retest reliability studies. Environ Resour Econ 12:357–374Google Scholar
  65. Mitchell RC, Carson RT (1989) Using surveys to value public goods: the contingent valuation method. Resources for the Future, Washington, DCGoogle Scholar
  66. Mkwara L, Marsh D, Scarpa R (2015). Testing the stability of welfare estimates in travel cost random utility models of recreation: an application to the Rotorua Lakes, New Zealand. 59th Australian Agricultural and Resource Economics Society ConferenceGoogle Scholar
  67. Moeltner K, Englin J (2004) Choice behavior under time-variant quality: state dependence versus Play-It-By-Ear in selecting ski resorts. J Bus Econ Stat 22(2):214–224Google Scholar
  68. Morrison MD, Blamey RK, Bennett JW (2000) Minimising payment vehicle bias in contingent valuation studies. Environ Resour Econ 16:407–422Google Scholar
  69. Murdock J (2006) Handling unobserved site characteristics in random utility models of recreation demand. J Environ Econ Manag 51(1):1–25Google Scholar
  70. Murphy JJ, Allen PG, Stevens TH, Weatherhead D (2005) A meta-analysis of hypothetial bias in stated preference valuation. Environ Resour Econ 30:313–325Google Scholar
  71. Parsons GR (2017) Travel cost. In: Chap PA, Boyle KJ, Brown TC (eds) A primer on nonmarket valuation, Ch. 6, Springer, New YorkGoogle Scholar
  72. Parsons GR, Stefanova S (2011) Gauging the value of the short-term site closures in a travel-cost RUM model of recreation demand with a little help from stated preference data. In: Whitehead J, Haab TC, Huang J (eds) Preference data for environmental valuation: combining revealed and stated preference approaches. Routledge, Taylor & Francis Group, New YorkGoogle Scholar
  73. Phaneuf DJ (2002) A random utility model for total maximum daily loads: estimating the benefits of watershed-based ambient water quality improvements. Water Resourc Res 38(11):1254Google Scholar
  74. Poe GL, Vossler CA (2011) Consequentiality and contingent values: an emerging paradigm. Edward Elgar, CheltenhamGoogle Scholar
  75. Randall A (1994) A difficulty with the travel cost method. Land Econ 70(1):88–96Google Scholar
  76. Reiling SD, Boyle KJ, Phillips ML, Anderson MW (1990) Temporal reliability of contingent values. Land Econ 66(2):128–134Google Scholar
  77. Roe BE, Just DR (2009) Internal and external validity in economics research: tradeoffs between experiments, field experiments, natural experiments, and field data. Am J Agric Econ 91(5):1266–1271Google Scholar
  78. Scarpa R, Thiene M, Train K (2008) Utility in willingness to pay space: a tool to address confounding random scale effects in destination choice to the Alps. Am J Agric Econ 90(4):994–1010Google Scholar
  79. Scott A (1965) The valuation of game resources: some theoretical aspects. In: Fisheries Canadian, Report, iv. Department of Fisheries of Canada, Ottawa, CanadaGoogle Scholar
  80. Shrestha RK, Loomis JB (2001) Testing a meta-analysis model for benefit transfer in international outdoor recreation. Ecol Econ 39(1):67–83Google Scholar
  81. Shrestha RK, Loomis JB (2003) Meta-analytic benefit transfer of outdoor recreation economic values: testing Out-of-Sample convergent validity. Environ Resour Econ 25(1):79–100Google Scholar
  82. Smith VK, Kaoru Y (1990) Signals or noise? Explaining the variation in recreation benefit estimates. Am J Agric Econ 72(2):419–433Google Scholar
  83. Stevens JB (1969) Effects of nonprice variables upon participation in water-oriented outdoor recreation: comment. Am J Agric Econ 51(1):192–193Google Scholar
  84. Stevens TH, DeCoteau NE, Willis CE (1997) Sensitivity of contingent valuation to alternative payment schedules. Land Econ 73(1):140–148Google Scholar
  85. Stumborg BE, Baerenklau KA, Bishop RC (2001) Nonpoint source pollution and present values: a contingent valuation study of Lake Mendota. Appl Econ Perspect Policy 23(1):120–132Google Scholar
  86. Tarrant MA, Manfredo MJ, Bayley PB, Hess R (1993) Effects of recall bias and nonresponse bias on self-report estimates of angling participation. North Am J Fish Manag 13(2):217–222Google Scholar
  87. Teisl MF, Boyle KJ, McCollum DW, Reiling SD (1995) Test-retest reliability of contingent valuation with independent sample pretest and posttest control groups. Am J Agric Econ 77:613–619Google Scholar
  88. Timmins C, Murdock J (2007) A revealed preference approach to the measurement of congestion in travel cost models. J Environ Econ Manag 53(2):230–249Google Scholar
  89. Trochim, William M (2002). The research methods knowledge base. Accessed 30 July 2017
  90. Trice AH, Wood SE (1958) Measurement of recreation benefits. Land Econ 34(3):195–207Google Scholar
  91. Vossler CA, Doyon M, Rondeau D (2012) Truth in consequentiality: theory and field evidence on discrete choice experiments. Am Econ J Microecon 4:145–171Google Scholar
  92. Vossler CA, Kerkvliet J (2003) A criterion validity test of the contingent valuation method: comparing hypothetical and actual voting behavior for a public referendum. J Environ Econ Manag 45:631–649Google Scholar
  93. Vossler CA, Kerkvliet J, Polasky S, Gainutdinova O (2003) Externally validating contingent valuation: an open space survey and referendum in Corvallis, Oregon. J Econ Behav Org 51:261–277Google Scholar
  94. Ward FA, Beal D (2000) Valuing nature with travel cost models. Edward Elgar, CheltenhamGoogle Scholar
  95. Whitehead J, Noonan DS, Marquardt E (2014) Criterion and predictive validity of revealed and stated preference data: the case of Mountain Home Music concert demand. Econ Bus Lett 3(2):87–95Google Scholar
  96. Zeller RA, Carmines EG (1980) Measurement in the social sciences: the link between theory and data. Cambridge University Press, CambridgeGoogle Scholar

Copyright information

© Springer Science+Business Media B.V., part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Agricultural and Applied EconomicsUniversity of Wisconsin-MadisonMadisonUSA
  2. 2.Virginia Tech Program in Real EstateBlacksburgUSA

Personalised recommendations