How to carry out conceptual properties norming studies as parameter estimation studies: Lessons from ecology

Behavior Research Methods volume 53, pages354–370(2021)Cite this article

Abstract

Conceptual properties norming studies (CPNs) ask participants to produce properties that describe concepts. From that data, different metrics may be computed (e.g., semantic richness, similarity measures), which are then used in studying concepts and as a source of carefully controlled stimuli for experimentation. Notwithstanding those metrics’ demonstrated usefulness, researchers have customarily overlooked that they are only point estimates of the true unknown population values, and therefore, only rough approximations. Thus, though research based on CPN data may produce reliable results, those results are likely to be general and coarse-grained. In contrast, we suggest viewing CPNs as parameter estimation procedures, where researchers obtain only estimates of the unknown population parameters. Thus, more specific and fine-grained analyses must consider those parameters’ variability. To this end, we introduce a probabilistic model from the field of ecology. Its related statistical expressions can be applied to compute estimates of CPNs’ parameters and their corresponding variances. Furthermore, those expressions can be used to guide the sampling process. The traditional practice in CPN studies is to use the same number of participants across concepts, intuitively believing that practice will render the computed metrics comparable across concepts and CPNs. In contrast, the current work shows why an equal number of participants per concept is generally not desirable. Using CPN data, we show how to use the equations and discuss how they may allow more reasonable analyses and comparisons of parameter values among different concepts in a CPN, and across different CPNs.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

US$ 39.95

Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. Anderson, J. R., & Fincham, M. (1996). Categorization and sensitivity to correlation. Journal of Experimental Psychology: Learning, Memory, & Cognition, 22, 259-277. https://doi.org/10.1037/0278-7393.22.2.259

    Article  Google Scholar 

  2. Ashby, F. G., & Alfonso-Reese, L. A. (1995). Categorization as probability density estimation. Journal of Mathematical Psychology, 39(2), 216-233. doi:https://doi.org/10.1006/jmps.1995.1021

    Article  Google Scholar 

  3. Ashcraft, M. H. (1978). Property norms for typical and atypical items from 17 categories: A description and discussion. Memory & Cognition, 6(3), 227-232. doi:https://doi.org/10.3758/BF03197450

    Article  Google Scholar 

  4. Barsalou, L.W. (1987). The instability of graded structure: Implications for the nature of concepts. In U. Neisser (Ed.), Concepts and conceptual development: Ecological and intellectual factors in categorization (pp. 101–140). Cambridge: Cambridge University Press.

    Google Scholar 

  5. Bolognesi, M., Pilgram, R., & van den Heerik, R. (2017). Reliability in content analysis: The case of semantic feature norms classification. Behavior Research Methods, 49(6), 1984–2001. doi:https://doi.org/10.3758/s13428-016-0838-6

    Article  PubMed  Google Scholar 

  6. Bruffaerts, R., De Deyne, S., Meersmans, K., Liuzzi, A. G., Storms, G., & Vandenberghe, R. (2019). Redefining the resolution of semantic knowledge in the brain: Advances made by the introduction of models of semantics in neuroimaging. Neuroscience and Biobehavioral Reviews, 103, 3-13. doi:https://doi.org/10.1016/j.neubiorev.2019.05.015

    Article  PubMed  Google Scholar 

  7. Buchanan, E.M., De Deyne, S., & Montefinese, M. (in press). A practical primer on processing semantic property norm data. Cognitive Processing doi: https://doi.org/10.1007/s10339-019-00939-6

  8. Canessa, E. & Chaigneau, S. E. (in press). Mathematical regularities of data from the property listing task. Journal of Mathematical Psychology, 102376. doi:https://doi.org/10.1016/j.jmp.2020.102376

  9. Chaigneau, S. E., Canessa, E., Barra, C., & Lagos, R. (2018). The role of variability in the property listing task. Behavior Research Methods, 50(3), 972-988. doi:https://doi.org/10.3758/s13428-017-0920-8

    Article  PubMed  Google Scholar 

  10. Chao, A., & Chiu, C. H. (2016). Species richness: Estimation and comparison. In N. Balakrishnan, T. Colton, B. Everitt, W. Piegorsch, F. Ruggeri & J. L. Teugels (Eds.) Wiley StatsRef: Statistics Reference Online (pp. 1–26). Chichester, UK: John Wiley & Sons, Ltd doi:https://doi.org/10.1002/9781118445112.stat03432.pub2.

    Google Scholar 

  11. Chao, A. & Jost, L. (2012). Coverage-based rarefaction: standardizing samples by completeness rather than by sample size. Ecology, 93, 2533–2547.

    Article  Google Scholar 

  12. Chao, A., Gotelli, N., Hsieh, T.C., Sander, E., Ma, K.H., Colwell, R. & Ellison, A. (2014). Rarefaction and extrapolation with Hill numbers: a framework for sampling and estimation in species diversity studies. Ecological Monographs, 84(1), 45–67.

    Article  Google Scholar 

  13. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. doi:https://doi.org/10.1177/001316446002000104

    Article  Google Scholar 

  14. Coley, J. D., Hayes, B., Lawson, C., & Moloney, M. (2004). Knowledge, expectations, and inductive reasoning within conceptual hierarchies. Cognition, 90(3), 217-253. doi:https://doi.org/10.1016/S0010-0277(03)00159-8

    Article  PubMed  Google Scholar 

  15. Cree, G. S., & McRae, K. (2003). Analyzing the factors underlying the structure and computation of the meaning of chipmunk, cherry, chisel, cheese, and cello (and many other such concrete nouns). Journal of Experimental Psychology: General, 132(2), 163-201. doi:https://doi.org/10.1037/0096-3445.132.2.163

    Article  Google Scholar 

  16. De Deyne, S., Navarro, D. J., Perfors, A., Brysbaert, M., & Storms, G. (2019). The “Small world of words” English word association norms for over 12,000 cue words. Behavior Research Methods, 51(3), 987-1006. doi:https://doi.org/10.3758/s13428-018-1115-7

    Article  PubMed  Google Scholar 

  17. Devereux, B. J., Tyler, L. K., Geertzen, J., & Randall, B. (2014). The centre for speech, language and the brain (CSLB) concept property norms. Behavior Research Methods, 46(4), 1119-1127. doi:https://doi.org/10.3758/s13428-013-0420-4

    Article  PubMed  Google Scholar 

  18. Devereux, B. J., Taylor, K. I., Randall, B., Geertzen, J., & Tyler, L. K. (2016). Feature statistics modulate the activation of meaning during spoken word processing. Cognitive Science, 40(2), 325-350. doi:https://doi.org/10.1111/cogs.12234

    Article  PubMed  Google Scholar 

  19. Duarte, L. R., Marquié, L., Marquié, J., Terrier, P., & Ousset, P. (2009). Analyzing feature distinctiveness in the processing of living and non-living concepts in Alzheimer's disease. Brain and Cognition, 71(2), 108–117. doi:https://doi.org/10.1016/j.bandc.2009.04.007

    Article  PubMed  Google Scholar 

  20. Flanagan, K. J., Copland, D. A., Chenery, H. J., Byrne, G. J., & Angwin, A. J. (2013). Alzheimer's disease is associated with distinctive semantic feature loss. Neuropsychologia, 51(10), 2016–2025. doi:https://doi.org/10.1016/j.neuropsychologia.2013.06.008

    Article  PubMed  Google Scholar 

  21. Garrard, P., Lambon Ralph, M. A., Hodges, J. R., & Patterson, K. (2001). Prototypicality, distinctiveness, and intercorrelation: Analyses of the semantic attributes of living and nonliving concepts, Cognitive Neuropsychology, 18(2), 125-174. doi:https://doi.org/10.1080/02643290125857

    Article  PubMed  Google Scholar 

  22. Glaser, B.G. & Strauss, A.L. (1967). The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago: Aldine.

    Google Scholar 

  23. Goldstone, R. (1994). Influences of categorization on perceptual discrimination. Journal of Experimental Psychology: General, 123(2), 178-200. doi:https://doi.org/10.1037/0096-3445.123.2.178

    Article  Google Scholar 

  24. Griffiths, T. L., Sanborn, A. N., Canini, K. R., & Navarro, D. J. (2008). Categorization as nonparametric Bayesian density estimation. In M. Oaksford and N. Chater (Eds.). The probabilistic mind: Prospects for rational models of cognition. Oxford: Oxford University Press.

    Google Scholar 

  25. Grondin, R., Lupker, S. J., & McRae, K. (2009). Shared features dominate semantic richness effects for concrete concepts. Journal of Memory and Language, 60(1), 1-19. doi:https://doi.org/10.1016/j.jml.2008.09.001

    Article  PubMed  PubMed Central  Google Scholar 

  26. Hallgren K. A. (2012). Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23–34. doi:https://doi.org/10.20982/tqmp.08.1.p023

    Article  PubMed  PubMed Central  Google Scholar 

  27. Hampton, J. A. (1979). Polymorphous concepts in semantic memory. Journal of Verbal Learning and Verbal Behavior, 18(4), 441-461. doi:https://doi.org/10.1016/S0022-5371(79)90246-9

    Article  Google Scholar 

  28. Hargreaves, I. S., & Pexman, P. M. (2014). Get rich quick: The signal to respond procedure reveals the time course of semantic richness effects during visual word recognition. Cognition, 131(2), 216–242. doi:https://doi.org/10.1016/j.cognition.2014.01.001

    Article  PubMed  Google Scholar 

  29. Hough, G., & Ferraris, D. (2010). Free listing: A method to gain initial insight of a food category. Food Quality and Preference, 21(3), 295-301. doi:https://doi.org/10.1016/j.foodqual.2009.04.001

    Article  Google Scholar 

  30. Jaccard, P. (1901). Distribution de la flore alpine dans le bassin des Dranses et dans quelques régions voisines. Bulletin de la Société Vaudoise des Sciences Naturelle, 37, 241–272.

    Google Scholar 

  31. Kounios, J., Green, D. L., Payne, L., Fleck, J. I., Grondin, R., & McRae, K. (2009). Semantic richness and the activation of concepts in semantic memory: Evidence from event-related potentials. Brain Research, 1282, 95–102. doi:https://doi.org/10.1016/j.brainres.2009.05.092

    Article  PubMed  PubMed Central  Google Scholar 

  32. Kremer, G., & Baroni, M. (2011). A set of semantic norms for German and Italian. Behavior Research Methods, 43(1), 97–109. doi:https://doi.org/10.3758/s13428-010-0028-x

    Article  PubMed  Google Scholar 

  33. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. doi:https://doi.org/10.2307/2529310

    Article  PubMed  Google Scholar 

  34. Lenci, A., Baroni, M., Cazzolli, G., & Marotta, G. (2013). BLIND: A set of semantic feature norms from the congenitally blind. Behavior Research Methods, 45(4), 1218-1233. doi:https://doi.org/10.3758/s13428-013-0323-4

    Article  PubMed  Google Scholar 

  35. McRae, K., Cree, G. S., Westmacott, R., & De Sa, V. R. (1999). Further evidence for feature correlations in semantic memory. Canadian Journal of Experimental Psychology, 53(4), 360–373. doi:https://doi.org/10.1037/h0087323

    Article  PubMed  Google Scholar 

  36. McRae, K., Cree, G. S., Seidenberg, M. S., & McNorgan, C. (2005). Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37(4), 547–559. doi:https://doi.org/10.3758/BF03192726

    Article  PubMed  Google Scholar 

  37. Montefinese, M., Ambrosini, E., Fairfield, B., & Mammarella, N. (2013). Semantic memory: A feature-based analysis and new norms for Italian. Behavior Research Methods, 45(2), 440-461. doi:https://doi.org/10.3758/s13428-012-0263-4

    Article  PubMed  Google Scholar 

  38. Perri, R., Zannino, G., Caltagirone, C., & Carlesimo, G. A. (2012). Alzheimer's disease and semantic deficits: A feature-listing study. Neuropsychology, 26(5), 652-663. doi:https://doi.org/10.1037/a0029302

    Article  PubMed  Google Scholar 

  39. Pexman, P. M., Hargreaves, I. S., Edwards, J. D., Henry, L. C., & Goodyear, B. G. (2007). The neural consequences of semantic richness: When more comes to mind, less activation is observed: Research report. Psychological Science, 18(5), 401–406. doi:https://doi.org/10.1111/j.1467-9280.2007.01913.x

    Article  PubMed  Google Scholar 

  40. Pexman, P. M., Hargreaves, I. S., Siakaluk, P. D., Bodner, G. E., & Pope, J. (2008). There are many ways to be rich: Effects of three measures of semantic richness on visual word recognition. Psychonomic Bulletin and Review, 15(1), 161-167. doi:https://doi.org/10.3758/PBR.15.1.161

    Article  PubMed  Google Scholar 

  41. Rasmussen, S.L., & Starr, N., (1979). Optimal and adaptive stopping in the search for new species. Journal of the American Statistical Association, 74, 661–667.

    Article  Google Scholar 

  42. Recchia, G., & Jones, M. N. (2012). The semantic richness of abstract concepts. Frontiers in Human Neuroscience, 6, 315. doi:https://doi.org/10.3389/fnhum.2012.00315

    Article  PubMed  PubMed Central  Google Scholar 

  43. Rosch, E., & Mervis, C. B. (1975). Family resemblances: Studies in the internal structure of categories. Cognitive Psychology, 7(4), 573–605. doi:https://doi.org/10.1016/0010-0285(75)90024-9

    Article  Google Scholar 

  44. Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., & Boyes-Braem, P. (1976). Basic objects in natural categories. Cognitive Psychology, 8(3), 382–439. https://doi.org/10.1016/0010-0285(76)90013-X

    Article  Google Scholar 

  45. Ruts, W., De Deyne, S., Ameel, E., Vanpaemel, W., Verbeemen, T., & Storms, G. (2004). Dutch norm data for 13 semantic categories and 338 exemplars. Behavior Research Methods, Instruments, & Computers, 36, 506–515. doi:https://doi.org/10.3758/BF03195597

    Article  Google Scholar 

  46. Schyns, P. G., Goldstone, R. L., & Thibaut, J. (1998). The development of features in object concepts. Behavioral and Brain Sciences, 21(1), 1–54. doi:https://doi.org/10.1017/S0140525X98000107

    Article  Google Scholar 

  47. Siew, C. S. Q. (in press). Feature distinctiveness effects in language acquisition and lexical processing: Insights from megastudies. Cognitive Processing doi:https://doi.org/10.1007/s10339-019-00947-6

  48. Sokal, R., & Michener, C. (1958). A statistical method for evaluating systematic relationships. University of Kansas Science Bulletin, 38, 1409–1438.

    Google Scholar 

  49. Tversky, A. (1977). Features of similarity. Psychological Review, 84(4), 327-352. doi:https://doi.org/10.1037/0033-295X.84.4.327

    Article  Google Scholar 

  50. Tversky, A., & Hutchinson, J. W. (1986). Nearest neighbor analysis of psychological spaces. Psychological Review, 93(1), 3–22. doi:https://doi.org/10.1037/0033-295X.93.1.3

    Article  Google Scholar 

  51. Vigliocco, G., Vinson, D. P., Lewis, W., & Garrett, M. F. (2004). Representing the meanings of object and action words: The featural and unitary semantic space hypothesis. Cognitive Psychology, 48(4), 422–488. doi:https://doi.org/10.1016/j.cogpsych.2003.09.001

    Article  PubMed  Google Scholar 

  52. Vinson, D. P., & Vigliocco, G. (2008). Semantic feature production norms for a large set of objects and events. Behavior Research Methods, 40(1), 183–190. doi:https://doi.org/10.3758/BRM.40.1.183

    Article  PubMed  Google Scholar 

  53. Vivas, J., Vivas, L., Comesaña, A., Coni, A. G., & Vorano, A. (2017). Spanish semantic feature production norms for 400 concrete concepts. Behavior Research Methods, 49(3), 1095–1106. doi:https://doi.org/10.3758/s13428-016-0777-2

    Article  PubMed  Google Scholar 

  54. Walker, L. J., & Hennig, K. H. (2004). Differing conceptions of moral exemplarity: Just, brave, and caring. Journal of Personality and Social Psychology, 86(4), 629–647. doi:https://doi.org/10.1037/0022-3514.86.4.629

    Article  PubMed  Google Scholar 

  55. Wiemer-Hastings, K., & Xu, X. (2005). Content Differences for Abstract and Concrete Concepts. Cognitive Science, 29, 719-736. doi:https://doi.org/10.1207/s15516709cog0000_33

    Article  Google Scholar 

  56. Wu, L., & Barsalou, L. W. (2009). Perceptual simulation in conceptual combination: Evidence from property generation. Acta Psychologica, 132(2), 173–189. doi:https://doi.org/10.1016/j.actpsy.2009.02.002

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This research was carried out with funds provided by ANID, Fondo Nacional de Desarrollo Científico y Tecnológico (FONDECYT) of the Chilean government grant 1200139. Felipe Medina thankfully acknowledges funding from Comisión Nacional de Investigación Científica y Tecnológica, CONICYT Ph.D. fellowship 21151523. We are grateful to Penny Pexman and Jorge Vivas for their valuable input on the current work. The authors declare to have no known conflicts of interest regarding the work being reported here.

None of the data or materials for the experiments reported here are available, and none of the experiments was preregistered. However, the interested reader may directly contact the corresponding author and he will send the corresponding materials and data to the reader.

Author information

Affiliations

  1. Center for Cognition Research (CINCO), School of Psychology, Universidad Adolfo Ibáñez, Av. Presidente Errázuriz 3328, Las Condes, Santiago, Chile

    Enrique Canessa & Sergio E. Chaigneau

  2. Faculty of Engineering and Science, Universidad Adolfo Ibáñez, Av. P. Hurtado 750, Lote H, Viña del Mar, Chile

    Enrique Canessa

  3. Center for Social and Cognitive Neuroscience, School of Psychology, Universidad Adolfo Ibáñez, Av. Presidente Errázuriz 3328, Las Condes, Santiago, Chile

    Sergio E. Chaigneau

  4. Programa de Bioestadística, Escuela de Salud Pública, Universidad de Chile, Santiago, Chile

    Rodrigo Lagos & Felipe A. Medina

  5. Instituto de Estadística, Facultad de Ciencias, Universidad de Valparaíso, Valparaíso, Chile

    Felipe A. Medina

Authors
  1. Enrique Canessa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sergio E. Chaigneau
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rodrigo Lagos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Felipe A. Medina
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Enrique Canessa.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Canessa, E., Chaigneau, S.E., Lagos, R. et al. How to carry out conceptual properties norming studies as parameter estimation studies: Lessons from ecology. Behav Res 53, 354–370 (2021). https://doi.org/10.3758/s13428-020-01439-8

Download citation

Keywords

  • Conceptual properties norming studies
  • Property listing task
  • Parameter estimation
  • Sample size determination
  • Sample coverage