Skip to main content
SpringerLink
Log in

Making a Wiser Crowd: Benefits of Individual Metacognitive Control on Crowd Performance

  • Published:
Computational Brain & Behavior Aims and scope Submit manuscript

Abstract

The wisdom of the crowd refers to the finding that judgments aggregated over individuals are typically more accurate than the average individual’s judgment. Here, we examine the potential for improving crowd judgments by allowing individuals to choose which of a set of queries to respond to. If individuals’ metacognitive assessments of what they know is accurate, allowing individuals to opt in to questions of interest or expertise has the potential to create a more informed knowledge base over which to aggregate. This prediction was confirmed: crowds composed of volunteered judgments were more accurate than crowds composed of forced judgments. Overall, allowing individuals to use private metacognitive knowledge holds much promise in enhancing judgments, including those of the crowd.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Bedford, T., & Cooke, R. (2001). Probabilistic risk analysis: foundations and methods. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Benjamin, A.S. (2008a). Memory is more than just remembering: strategic control of encoding, accessing memory, and making decisions. Psychology of learning and motivation, 48, 175–223.

    Article  Google Scholar 

  • Benjamin, A.S., & Ross, B.H. (2008b). The psychology of learning and motivation: skill and strategy in memory use Vol. 48. New York: Academic Press.

    Google Scholar 

  • Brewer, W.F., & Sampaio, C. (2012). The metamemory approach to confidence: a test using semantic memory. Journal of Memory and Language, 67(1), 59–77.

    Article  Google Scholar 

  • Budescu, D., & Chen, E. (2014). Identifying expertise to extract the wisdom of crowds. Management Science, 61, 267–280.

    Article  Google Scholar 

  • Culpepper, S.A., & Balamuta, J.J. (2017). A hierarchical model for accuracy and choice on standardized tests. Psychometrika, 82(3), 820–845.

    Article  Google Scholar 

  • Fiechter, J.L., Benjamin, A.S., Unsworth, N. (2016). 16 the metacognitive foundations of effective remembering, (p. 307). Oxford: The Oxford Handbook of Metamemory.

    Google Scholar 

  • Galton, F. (1907). Vox populi. Nature, 75(7), 450–451.

    Article  Google Scholar 

  • Goldsmith, M., & Koriat, A. (2007). The strategic regulation of memory accuracy and informativeness. Psychology of learning and motivation, 48, 1–60.

    Article  Google Scholar 

  • Hsu, C.-C., & Sandford, B.A. (2007). The delphi technique: making sense of consensus. Practical assessment, research & evaluation, 12(10), 1–8.

    Google Scholar 

  • Jarosz, A.F., & Wiley, J. (2014). What are the odds? a practical guide to computing and reporting bayes factors. The Journal of Problem Solving, 7(1), 2.

    Article  Google Scholar 

  • Jeffreys, H. (1961). Theory of Probability. Oxford.

  • Kepecs, A., & Mainen, Z.F. (2012). A computational framework for the study of confidence in humans and animals. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 367(1594), 1322–1337.

    Article  PubMed  Google Scholar 

  • Koriat, A., & Goldsmith, M. (1996). Monitoring and control processes in the strategic regulation of memory accuracy. Psychological Review, 103(3), 490.

    Article  PubMed  Google Scholar 

  • Kornell, N., & Metcalfe, J. (2006). Study efficacy and the region of proximal learning framework. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32(3), 609.

    PubMed  Google Scholar 

  • Kunimoto, C., Miller, J., Pashler, H. (2001). Confidence and accuracy of near-threshold discrimination responses. Consciousness and Cognition, 10(3), 294–340.

    Article  PubMed  Google Scholar 

  • Lee, M.D., Steyvers, M., de Young, M., Miller, B.J. (2012). Inferring expertise in knowledge and prediction ranking tasks. Topics in Cognitive Science, 4, 151–163.

    Article  PubMed  Google Scholar 

  • Lee, M.D., & Danileiko, I. (2014). Using cognitive models to combine probability estimates. Judgment and Decision Making, 9(3), 259.

    Google Scholar 

  • Love, J., Selker, R., Marsman, M., Jamil, T., Dropmann, D., Verhagen, A., Wagenmakers, E. (2018). Jasp (version 0.8.6). Computer software. Retrieved from https://jasp-stats.org.

  • Mannes, A.E., Soll, J.B., Larrick, R.P. (2014). The wisdom of select crowds. Journal of Personality and Social Psychology, 107(2), 276.

    Article  PubMed  Google Scholar 

  • Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S.E., Moore, D., Atanasov, P., Swift, S.A., Murray, T., Stone, E., Tetlock, P.E. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science, 25(5), 1106–1115. PMID: 24659192.

    Article  PubMed  Google Scholar 

  • Merkle, E.C., Steyvers, M., Mellers, B., Tetlock, P.E. (2016). Item response models of probability judgments: application to a geopolitical forecasting tournament. Decision, 3(1), 1.

    Article  Google Scholar 

  • Merkle, E.C., Steyvers, M., Mellers, B., Tetlock, P.E. (2017). A neglected dimension of good forecasting judgment: the questions we choose also matter. International Journal of Forecasting, 33(4), 817–832.

    Article  Google Scholar 

  • Middlebrooks, P.G., & Sommer, M.A. (2011). Metacognition in monkeys during an oculomotor task. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(2), 325.

    PubMed  Google Scholar 

  • Olson, K.C., & Karvetsi, C.W. (2013). Improving expert judgment by coherence weighting. In proceedings of 2013 IEEE International Conference on Intelligence and Security Informatics.

  • Rocklin, T.R. (1994). Self-adapted testing. Applied Measurement in Education, 7(1), 3–14.

    Article  Google Scholar 

  • Rouder, J.N., Speckman, P.L., Sun, D., Morey, R.D., Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225–237.

    Article  Google Scholar 

  • Rouder, J.N. (2018). Bayes factor calculator. Website. Retrieved from http://pcl.missouri.edu/bayesfactor. Accessed: 2018-04-23.

  • Rowe, G., & Wright, G. (1999). The delphi technique as a forecasting tool: issues and analysis. International Journal of Forecasting, 15(4), 353–375.

    Article  Google Scholar 

  • Schraw, G., Flowerday, T., Reisetter, M.F. (1998). The role of choice in reader engagement. Journal of Educational Psychology, 90(4), 705.

    Article  Google Scholar 

  • Sniezek, J.A. (1989). An examination of group process in judgmental forecasting. International Journal of Forecasting, 5(2), 171– 178.

    Article  Google Scholar 

  • Steyvers, M., & Miller, B. (2015). Cognition and collective intelligence. In Bernstein, M., & Malone, T. W. (Eds.) Handbook of Collective Intelligence (pp. 119–138): MIT Press.

  • Steyvers, M., Miller, B., Hemmer, P., Lee, M.D. (2009). The wisdom of crowds in the recollection of order information. In Advances in neural information processing systems (pp 1785–1793).

  • Surowiecki, J. (2004). The wisdom of crowds: why the many are smarter than the few and how collective wisdom shapes business, economics society and nations. Brown: Little.

    Google Scholar 

  • Tullis, J.G., & Benjamin, A.S. (2011). On the effectiveness of self-paced learning. Journal of Memory and Language, 64(2), 109–118.

    Article  PubMed  PubMed Central  Google Scholar 

  • Turner, B.M., Steyvers, M., Merkle, E.C., Budescu, D.V., Wallsten, T.S. (2014). Forecast aggregation via recalibration. Machine Learning, 95(3), 261–289.

    Article  Google Scholar 

  • Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14(5), 779–804.

    Article  Google Scholar 

  • Weiss, D.J., Brennan, K., Thomas, R., Kirlik, A., Miller, S.M. (2009). Criteria for performance evaluation. Judgment and Decision Making, 4, 164–174.

    Google Scholar 

  • Wixted, J.T., Mickes, L., Clark, S.E., Gronlund, S.D., Roediger, III H.L. (2015). Initial eyewitness confidence reliably predicts eyewitness identification accuracy. American Psychologist, 70(6), 515.

    Article  PubMed  Google Scholar 

  • Yi, S.K.M., Steyvers, M., Lee, M.D., Dry, M.J. (2012). The wisdom of the crowd in combinatorial problems. Cognitive Science, 36(3), 452–470.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of California, Irvine, USA

    Stephen T. Bennett, Percy K. Mistry & Mark Steyvers

  2. University of Illinois at Urbana-Champaign, Illinois, USA

    Aaron S. Benjamin

Authors
  1. Stephen T. Bennett
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aaron S. Benjamin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Percy K. Mistry
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mark Steyvers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stephen T. Bennett.

Appendix

Appendix

Experiment 1b judgments

Fig. 4
figure 4

Question responses for the self-directed and control participants in the hard condition with questions sorted by the number of participants who selected the question in the self-directed condition. Green squares represent correct responses, red squares represent incorrect responses, and white squares represent no response. Self-directed participants tend to cluster around the same collection of questions when compared to control participants

Full size image

Experiment 2a judgments

Fig. 5
figure 5

Question responses from the partial opt-in and control conditions with questions sorted by the number of self-directed participants who selected the question. Green squares represent correct responses, red squares represent incorrect responses, and white squares represent no response. Self-directed participants tend to cluster around the same collection of questions when compared to control participants

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bennett, S.T., Benjamin, A.S., Mistry, P.K. et al. Making a Wiser Crowd: Benefits of Individual Metacognitive Control on Crowd Performance. Comput Brain Behav 1, 90–99 (2018). https://doi.org/10.1007/s42113-018-0006-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42113-018-0006-4

Keywords

Search

Navigation