International Journal of Biometeorology

, Volume 59, Issue 7, pp 917–926 | Cite as

Assessing accuracy in citizen science-based plant phenology monitoring

  • Kerissa K. FuccilloEmail author
  • Theresa M. Crimmins
  • Catherine E. de Rivera
  • Timothy S. Elder
Original Paper


In the USA, thousands of volunteers are engaged in tracking plant and animal phenology through a variety of citizen science programs for the purpose of amassing spatially and temporally comprehensive datasets useful to scientists and resource managers. The quality of these observations and their suitability for scientific analysis, however, remains largely unevaluated. We aimed to evaluate the accuracy of plant phenology observations collected by citizen scientist volunteers following protocols designed by the USA National Phenology Network (USA-NPN). Phenology observations made by volunteers receiving several hours of formal training were compared to those collected independently by a professional ecologist. Approximately 11,000 observations were recorded by 28 volunteers over the course of one field season. Volunteers consistently identified phenophases correctly (91 % overall) for the 19 species observed. Volunteers demonstrated greatest overall accuracy identifying unfolded leaves, ripe fruits, and open flowers. Transitional accuracy decreased for some species/phenophase combinations (70 % average), and accuracy varied significantly by phenophase and species (p < 0.0001). Volunteers who submitted fewer observations over the period of study did not exhibit a higher error rate than those who submitted more total observations. Overall, these results suggest that volunteers with limited training can provide reliable observations when following explicit, standardized protocols. Future studies should investigate different observation models (i.e., group/individual, online/in-person training) over subsequent seasons with multiple expert comparisons to further substantiate the ability of these monitoring programs to supply accurate broadscale datasets capable of answering pressing ecological questions about global change.


Citizen science Data accuracy Phenology Protocols Natures Notebook USA National Phenology Network 



Sincere thanks to all contributors to this study including Lori Anne Barnett, Ellen Denny, Alyssa Rosemartin, Jake Weltzin, and USA-NPN National Coordinating Office staff, Dr. Marion Dresner, Dr. Sybil Kelly, Kelly Fisher, Jill Van Winkle, and Rhiannon LeFay from Portland State University, all volunteers from Portland Budwatch (with a special thanks to Harriet Denison and Mark Feldman for their dedication to the project), and Kendra Peterson-Morgan and Rachel Feliz of Portland Parks and Recreation. The USA-NPN National Coordinating Office is grateful for the critical role this study played in refining plant phenology protocols. This project was funded in part by NSF award #0941763, Interactive Teaching Materials for Understanding Ecological Response from Climate Change in Urban Forests.

Supplementary material

484_2014_892_MOESM1_ESM.pdf (234 kb)
ESM 1 (PDF 234 kb)


  1. Bas Y, Devictor V, Moussus JP, Jiguet F (2008) Accounting for weather and time-of-day parameters when analyzing count data from monitoring programs. Biodivers Conserv 17:3403–3416CrossRefGoogle Scholar
  2. Beaubien EG, Hamann A (2011) Plant phenology networks of citizen scientists: recommendations from two decades of experience in Canada. Int J Biometeorol 55:833–841CrossRefGoogle Scholar
  3. Bell JJ (2007) The use of volunteers for conducting sponge biodiversity assessments and monitoring using a morphological approach on Indo-Pacific coral reefs. Aquat Conserv 17:133–145CrossRefGoogle Scholar
  4. Both C, van Asch M, Bijlsma RG, van den Burg AB, Visser ME (2009) Climate change and unequal phenological changes across four trophic levels: constraints or adaptations? J Anim Ecol 78:73–83CrossRefGoogle Scholar
  5. Boudreau SA, Yan ND (2004) Auditing the accuracy of a volunteer-based surveillance program for an aquatic invader Bythotrephes. Environ Monit Assess 91:17–26CrossRefGoogle Scholar
  6. Brossard D, Lewenstein B, Bonney R (2005) Scientific knowledge and attitude change: the impact of a citizen science project. Int J Sci Educ 27:1099–1121CrossRefGoogle Scholar
  7. Cleland EE, Chuine I, Menzel A, Mooney HA, Schwartz MD (2007) Shifting plant phenology in response to global change. Trends Ecol Evol 22:357–365CrossRefGoogle Scholar
  8. Cooper CB (2013) Is there a weekend bias in clutch-initiation dates from citizen science? Implications for studies of avian breeding phenology. Int J Biometeorol 58:1415–1419Google Scholar
  9. Cooper CB, Dickinson J, Phillips TB, Bonney R (2007) Citizen science as a tool for conservation in residential ecosystems. Ecol Soc 12:11Google Scholar
  10. Couter JR, Johnson RJ, Stuyck CM, Lang BA, Kaiser EW (2012) Weekend bias in citizen science data reporting: implications for phenology studies. Int J Biometeorol 57:715–720CrossRefGoogle Scholar
  11. Crall, AW, Newman GJ, Stohlgren TJ, Holfelder KA, Graham J, DM Waller (2011) Assessing citizen science data quality: an invasive species case study. Conservation Letters 0:1–10Google Scholar
  12. Delaney DG, Sperling CD, Adams CS, Leung B (2008) Marine invasive species: validation of citizen science and implications for national monitoring networks. Biol Invasions 10:117–128CrossRefGoogle Scholar
  13. Dickinson JL, Zuckerberg B, Bonter DN (2010) Citizen science as an ecological research tool: challenges and benefits. Annu Rev Ecol Evol Syst 4:149–172CrossRefGoogle Scholar
  14. Edwards M, Richardson AJ (2004) Impact of climate change on marine pelagic phenology and trophic mismatch. Nature 430:881–884CrossRefGoogle Scholar
  15. Fore LS, Paulsen K, O’Laughlin K (2001) Assessing the performance of volunteers in monitoring streams. Freshwater Biol 46(1):109–123CrossRefGoogle Scholar
  16. Foster-Smith J, Evans SM (2003) The value of marine ecological data collected by volunteers. Biol Conserv 113:199–213CrossRefGoogle Scholar
  17. Galloway AWE, Tudor MT, Vander Haegen WM (2006) The reliability of citizen science: a case study of Oregon white oak stand surveys. Wildlife Soc B 34:1425–1429CrossRefGoogle Scholar
  18. Gardiner MM, Allee LL, Brown PMJ, Losey JE, Roy HE, Smyth RR (2012) Lessons from lady beetles: accuracy of monitoring data from US and UK citizen-science programs. Front Ecol Environ 10:471–476CrossRefGoogle Scholar
  19. Greenwood J (2007) Citizens, science and bird conservation. J Ornithol 148:77–124CrossRefGoogle Scholar
  20. Haklay M (2010) How good is volunteered geographical information? A comparative study of OpenStreetMap and ordnance survey datasets. Environ Plann B 37:682–703CrossRefGoogle Scholar
  21. Ingwell LI, Preisser EL (2010) Using citizen science programs to identify host resistance in pest-invaded forests. Conserv Biol 25:182–188CrossRefGoogle Scholar
  22. Jiguet F (2009) Method learning caused by a first-time observer effect in a newly started breeding bird survey. Bird Study 56:253–258CrossRefGoogle Scholar
  23. Koch E (2010) Global framework for data collection—data bases, data availability, future networks, online databases. In: Hudson IL, Keatley MR (eds) Phenological research: methods for environmental and climate change analysis. Springer, Dordrecht, pp 23–61CrossRefGoogle Scholar
  24. Kremen C, Ullmann KS, Thorp RW (2010) Evaluating the quality of citizen-scientist data on pollinator communities. Conserv Biol 25:607–617CrossRefGoogle Scholar
  25. Lovell S, Hamer M, Slotow R, Herbert D (2009) An assessment of the use of volunteers for terrestrial invertebrate biodiversity surveys. Biodivers Conserv 18:3295–3307CrossRefGoogle Scholar
  26. Memmott J, Craze PG, Waser NM, Price MV (2007) Global warming and the disruption of plant-pollinator interactions. Ecol Lett 10:710–717CrossRefGoogle Scholar
  27. Menzel A, Sparks TH, Estrella N, Koch E, Aasa A, Ahas R, Alm-Kubler K, Bissolli P, Braslavska O, Briede A et al (2006) European phenological response to climate change matches the warming pattern. Glob Chang Biol 12:1969–1976Google Scholar
  28. Nerbonne JF, Vondracek B (2003) Volunteer macroinvertebrate monitoring: assessing training needs through examining error and bias in untrained volunteers. J N Am Benthol Soc 22:152–163CrossRefGoogle Scholar
  29. Parmesan C, Yohe G (2003) A globally coherent fingerprint of climate change impacts across natural systems. Nature 421:37–42CrossRefGoogle Scholar
  30. Pearce-Higgins JW, Yalden DW, Whittingham MJ (2005) Warmer springs advance the breeding phenology of golden plovers Pluvialis apricaria and their prey (Tipulidae). Oecologia 143:470–476CrossRefGoogle Scholar
  31. Root TL, Price JT, Hall KR, Schneider SH, Rosenzweig C, Pounds JA (2003) Fingerprints of global warming on wild animals and plants. Nature 421:57–60CrossRefGoogle Scholar
  32. Schmeller DS, Henry P-Y, Julliard R, Gruber B, Clobert J, Dziock F, Lengyel S, Nowicki P, Déri E, Budrys E, Kull T, Tali K, Bauch B, Settele J, Van Swaay C, Kobler A, Babij V, Papastergiadou E, Henle K (2009) Advantages of volunteer-based biodiversity monitoring in Europe. Conserv Biol 23:307–316CrossRefGoogle Scholar
  33. Siebeck UE, Marshall NJ, Klüter A, Hoegh-Guldberg O (2006) Monitoring coral bleaching using a colour reference card. Coral Reefs 25:453–460CrossRefGoogle Scholar
  34. Sparks TH, Huber K, Tryianowski P (2008) Something for the weekend? Examining the bias in avian phenological recording. Int J Biometeorol 52:505–510CrossRefGoogle Scholar
  35. Thackeray SJ, Sparks TH, Frederiksen M, Burthe S, Bacon PJ, Bell JR, Botham MS, Brereton TM, Bright PW, Carvalho L, Clutton-Brock T, Dawson A, Edwards M, Elliott JM, Harrington R, Johns D, Jones ID, Jones JT, Leech DI, Roy DB, Scott WA, Smith M, Smithers RJ, Winfield IJ, Wanless S (2010) Trophic level asynchrony in rates of phenological change for marine, freshwater and terrestrial environments. Glob Change Biol 16:3304–3313CrossRefGoogle Scholar
  36. Tregidgo, DJ, SE West, and MR Ashmore (2014) Can citizen science produce good science? Testing the OPAL Air Survey methodology, using lichens as indicators of nitrogenous pollution. Environ Pollut 182:448–451Google Scholar
  37. Trumbull D, Bonney R, Bascom D, Cabral A (2000) Thinking scientifically during participation in a citizen-science project. Sci Educ 84:265–275CrossRefGoogle Scholar
  38. Walther G, Post E, Convey P, Menzel A, Parmesan C, Beebee TJC, Fromentin J-M, Hoegh-Guldberg O, Bairlein F (2002) Ecological responses to recent climate changes. Nature 416:389–395CrossRefGoogle Scholar
  39. Winder M, Schindler D (2004) Climate change uncouples trophic interactions in an aquatic system. Ecology 85:2100–2106CrossRefGoogle Scholar
  40. Wolkovich EM, Cook BI, Allen JM, Crimmins TM, Travers S, Pau S, Regetz J, Davies TJ, Betancourt JL, Kraft NJB, Ault TR, Bolmgren K, Mazer SJ, McCabe GJ, McGill BJ, Parmesan C, Salamin N, Schwartz MD, Cleland EE (2012) Warming experiments underpredict plant phenological responses to climate change. Nature 485:494–497Google Scholar

Copyright information

© ISB 2014

Authors and Affiliations

  • Kerissa K. Fuccillo
    • 1
    Email author
  • Theresa M. Crimmins
    • 2
  • Catherine E. de Rivera
    • 1
  • Timothy S. Elder
    • 1
  1. 1.Department of Environmental Science and ManagementPortland State UniversityPortlandUSA
  2. 2.USA National Phenology NetworkNational Coordinating OfficeTucsonUSA

Personalised recommendations