Abstract
In empirical research certain measurements are frequently performed. Measurement is the assigning of numbers to subjects to denote a property. These measurements take place at a level of measurement: nominal, ordinal, interval or higher. Especially in the social sciences, the assignment to a category is very often based on the judgement of an expert and not on an objective and clear criterium. Still it is desirable that the ‘scientists agree’, i.e. the assignment will not or scarcely differ if another rater had performed the assignment task. So it is of importance that there is agreement among the raters. There is agreement when raters assign an entity to the same category. Agreement is not only investigated within the social sciences, but in general also in the fields of biology and medicine.
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
ANDERBERG, M. R., Cluster Analysis for Applications (New York: Academic Press, 1973).
ARMITAGE, P., BLENDIS, L. M. and SMYLLIE, H. C., ‘The Measurement of Observer Disagreement in the Recording of Signs’, Journal of the Royal Statistical Society (A), 129 (1966) 98–109.
BARTKO, J. J. and CARPENTER, W. T., ‘Methods and Theory of Reliability’, Journal of Nervous and Mental Disease, 163 (1976) 307–17.
BENNETT, E. M., BLOMQUIST, R. L. and GOLDSTEIN, A. C, ‘Response Stability in Limited Response Questioning’, Public Opinion Quarterly, 18 (1954) 218–23.
BRENNAN, R. L. and LIGHT, R. J., ‘Measuring Agreement when Two Observers Classify People into Categories not Defined in Advance’, British Journal of Mathematical and Statistical Psychology, 27 (1974) 154–63.
CARTWRIGHT, D. S., ‘A Rapid Non-parametric Estimate of Multijudge Reliability’, Psychometrika, 21 (1956) 17–29.
CICCHETTI, D. V., ‘A New Measure of Agreement Between Rank-Ordered Variables’, American Psychological Association. Proceedings of the 80th Annual Convention, 7 (1972) 17–18.
CLEMENT, P. G., ‘A Formula for Computing Interobserver Agreement’, Psychological Reports, 39 (1976) 257–8.
COHEN, J., ‘A Coefficient of Agreement for Nominal Scales’, Educational and Psychological Measurement, 20 (1960) 37–46.
CRITTENDEN, K. S. and HILL, R. J., ‘Coding Reliability and Validity of Interview Data’, American Sociological Review, 36 (1971) 1073–80.
DICE, L. R., ‘Measures of the Amount of Ecological Association between Species’, Ecology, 26 (1945) 297–302.
ELSTON, R. C., SCHROEDER, S. R. and ROJAHN, J., ‘Measures of Observer Agreement when Binomial Data are Collected in Free Operant Situations’, Journal of Behavioral Assessment, 4 (1982) 299–310.
FLANDERS, N. A., ‘Estimating Reliability’, in AMIDON, E. J. and HOUGH, J. B. (eds) Interaction Analysis: Theory, Research and Applications (Reading, Mass.: Addison-Wesley, 1967) 161–6.
FLEISS, J. L., ‘Estimating the Accuracy of Dichotomous Judgements’, Psychometrika, 30 (1965) 469–79.
—— ’Measuring Nominal Scale Agreement Among many Raters’, Psychological Bulletin, 76 (1971) 378–82.
GALTUNG, J., ‘Measurement of Agreement’, in GALTUNG, J. (ed.), Papers on Methodology. Theory and Methods of Social Research, vol. II (Copenhagen: Christian Eijlers, 1979) 82–135.
GARRETT, C. S., ‘Modification of the Scott Coefficient as an Observer Agreement Estimate for Marginal-form Observation Scale Data’, Journal of Experimental Education, 43 (1975) 4, 21–6.
GOODMAN, L. A. and KRUSKAL, W. H., ‘Measures of Association for Cross-classifications’, Journal of the American Statistical Association, 49 (1954) 732–64.
HARRIS, F. C. and LAHEY, B. B., ‘A Method for Combining Occurrence and Non-occurrence Interobserver Agreement Scores’, Journal of Applied Behavior Analysis, 10 (1978) 523–7.
HAWKINS, R. P. and DOTSON, V. A., ‘Reliability Scores that delude: An Alice in Wonderland Trip Through the Misleading Characteristics of Interobserver Agreement Scores in Interval Recording’, in RAMP, E. and SEMB, G. (eds) Behavior Analysis: Areas of Research and Application (Englewood Cliffs, N.J.: Prentice-Hall, 1975) 359–76.
HOLLEY, J. W. and GUILFORD, J. P., ‘A Note on the G-index of Agreement’, Educational and Psychological Measurement, 24 (1964) 749–53.
HOPKINS, B. L. and HERMANN, J. A., ‘Evaluating Interobserver Reliability of Interval Data’, Journal of Applied Behavior Analysis, 10 (1977) 121–6.
HOUSE, A. E., HOUSE, B. J. and CAMPBELL, M. B., ‘Measures of Interobserver Agreement: Calculation Formulas and Distribution Effects’, Journal of Behavioral Assessment, 3 (1981) 37–57.
HUBERT, L. J., ‘Nominal Scale Response Agreement as a Generalized Correlation’, British Journal of Mathematical and Statistical Psychology, 30 (1977) 98–103.
JANES, C. L., ‘Extension of the Random Error Coefficient of Agreement to N * N tables’, British Journal of Psychiatry, 134 (1979) 617–19.
JANSON, S. and VEGELIUS, J., ‘On the Generalization of the G-index and the Phi Coefficient to Nominal Scales’, Multivariate Behavioral Research, 14 (1979) 255–69.
—— and —— ‘The J-index as a Measure of Nominal Scale Response Agreement’, Applied Psychological Measurement, 6 (1982) 111–21.
KENT, R. N. and FOSTER, S. L., ‘Direct Observational Procedures: Methodological Issues in Naturalistic Settings’, in CIMINERO, A. R., CALHOUN, K. S. and ADAMS, H. E. (eds) Handbook of Behavioral Assessment (New York: Wiley, 1977) 279–328.
KRIPPENDORFF, K., Content Analysis. An Introduction to its Methodology (Beverly Hills: Sage, 1981).
LIGHT, R. J., ‘Issues in the Analysis of Categorical Data’, in TRAVERS, R. M. W. (ed.) Second Handbook of Research Teaching (Chicago: Rand-McNally, 1973) 318–81.
LISCH, R. and KRIZ, J., Grundlagen und Modelle der Inhaltanalyse (Reinbek: Rororo, 1978).
MAXWELL, A. E., ‘Coefficients of Agreement Between Observers and their Interpretation’, British Journal of Psychiatry, 130 (1977) 79–83.
MAXWELL, A. E. and PILLINER, A. E. G., ‘Deriving Coefficients of Reliability and Agreement for Ratings’, British Journal of Mathematical and Statistical Psychology, 21 (1968) 105–16.
MOKKEN, R. J., A Theory and Procedure of Scale Analysis: With Applications in Political Research (The Hague: Mouton, 1971).
MONTGOMMERY, A. C. and CRITTENDEN, K. S., ‘Improving Coding Reliability for Open-ended Questions’, Public Opinion Quarterly, 41 (1977) 235–43.
POPPING, R., ‘Traces of Agreement: On the Dot-product as a Coefficient of Agreement’, Quality and Quantity, 17 (1983a) 1–18.
POPPING, R., ‘Overeenstemmingsmaten voor Nominale Data’, unpublished PhD thesis, University of Groningen (1983b).
POPPING, R., ‘AGREE, a package for Computing Nominal Scale Agreement’, Computational Statistics and Data Analysis, 2 (1984) 182–5.
RAE, D. W. and TAYLOR, M., The Analysis of Political Cleavages (New Haven: Yale U.P., 1970) 115–45.
Editor information
Editors and Affiliations
Copyright information
© 1988 Willem E. Saris and Irmtraud N. Gallhofer
About this chapter
Cite this chapter
Popping, R. (1988). On Agreement Indices for Nominal Data. In: Saris, W.E., Gallhofer, I.N. (eds) Sociometric Research. Palgrave Macmillan, London. https://doi.org/10.1007/978-1-349-19051-5_6
Download citation
DOI: https://doi.org/10.1007/978-1-349-19051-5_6
Publisher Name: Palgrave Macmillan, London
Print ISBN: 978-1-349-19053-9
Online ISBN: 978-1-349-19051-5
eBook Packages: Palgrave Social & Cultural Studies CollectionSocial Sciences (R0)