Factors Influencing the Reliability and Validity of Observation Data

  • Theodore Jacob
  • Daniel L. Tennenbaum
  • Gloria Krahn
Part of the Applied Clinical Psychology book series (NSSB)

Abstract

During the past 3 decades, researchers and clinicians alike have become increasingly interested in the nature of family interaction in their attempts to understand, modify, and prevent such diverse maladaptations as schizophrenia, depression, alcoholism, and childhood aggression. Overall, such investigations have sought to (a) construct and evaluate theoretical models of marital and parent—child behavior with the aim of understanding the development and functioning of family systems; (b) identify family interactions that reliably differentiate problem from nonproblem families as a first step toward developing more effective methods of treatment and prevention; and (c) assess the influence of various treatment approaches on the nature of intrafamilial interactions.

Keywords

Percentage Agreement Interobserver Reliability Behavioral Assessment Apply Behavior Analysis Family Interaction 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Anthony, N. (1971). Comparisons of clients’ standard, exaggerated, and matching MMPI profiles. Journal of Consulting and Clinical Psychology, 27, 253–256.Google Scholar
  2. Aragona, J., and Eyberg, S. (1981). Neglected children: Mother’s report of child behavior problems and observed verbal behavior. Child Development, 52, 596–602.PubMedCrossRefGoogle Scholar
  3. Arrington, E. (1943). Time sampling in studies of social behavior: A critical review of technique and results with research suggestions. Psychological Bulletin, 40, 81–124.CrossRefGoogle Scholar
  4. Bales, R. F. (1950). Interaction Process Analysis. New York: Addison-Wesley.Google Scholar
  5. Bell, R. Q. (1964). Structuring parent—child interaction for direct observation. Child Development, 35, 1009–1020.PubMedGoogle Scholar
  6. Belsky, J. ( 1977, March). Mother—infant interaction at home and in the laboratory: The effect of context. Paper presented at the Society for Research in Child Development conference, New Orleans.Google Scholar
  7. Bernal, M. F., Gibson, D. M., Williams, D. E., and Pesses, D. I. (1971). A device for automatic audio tape recording. Journal of Applied Behavior Analysis, 4, 151–156.PubMedCrossRefGoogle Scholar
  8. Blurton Jones, N. G., and Woodson, R. H. (1979). Describing behavior: The ethologists’ perspective. In M. E. Lamb, S. J. Suomi, and G. R. Stephenson (Eds.), Social Interaction Analysis. Madison: University of Wiscons in Press.Google Scholar
  9. Borduin, C. M., and Henggeler, S. W. (1981). Social class, experimental setting, and task characteristics as determinants of mother-child interaction. Developmental Psychology, 17, 209–214.CrossRefGoogle Scholar
  10. Bronfenbrenner, U. (1974). Developmental research, public policy, and the ecology of childhood. Child Development, 45, 1–5.CrossRefGoogle Scholar
  11. Brookhart, J., and Hock, E. (1976). The effects of experimental context and experimental background on infants’ behavior toward their mothers and a stranger. Child Development, 47, 333–340.CrossRefGoogle Scholar
  12. Campbell, D. T., and Stanley, J. C. (1963). Experimental and quasi-experimental designs for research and teaching. In N. L. Gage (Ed.), Handbook of research on teaching. Chicago: Rand McNally.Google Scholar
  13. Canter, F. (1963). Simulation of the California Personality Inventory and the adjustment of the simulator. Journal of Consulting Psychology, 27, 253–256.PubMedCrossRefGoogle Scholar
  14. Carroll, J. B. (1961). The nature of the data, or how to choose a correlation coefficient. Psychometrika, 26, 347–372.CrossRefGoogle Scholar
  15. Christensen, A., and Hazzard, A. (1983). Reactive effects during naturalistic observation of families. Behavioral Assessment, 5, 349–362.Google Scholar
  16. Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220.PubMedCrossRefGoogle Scholar
  17. Cohen, J. (1969). Statistical power analysis for the behavioral sciences. New York: Academic Press.Google Scholar
  18. Cohen, J. (1972). Weighted chi square: An extension of the kappa method. Educational and Psychological Measurement, 32, 61–74.CrossRefGoogle Scholar
  19. Cone, J. D. (1977). The relevance of reliability and validity for behavioral assessment. Behavior Therapy, 8, 411–426.CrossRefGoogle Scholar
  20. Conte, J. R. (1979). An experimental investigation of subject reactivity to observation of video camera and human observer (Doctoral dissertation, University of Washington). Dissertation Abstracts International, 40 (6-A), 3535A.Google Scholar
  21. Cronbach, L. J., Gleser, G. C., Nanda, H., and Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.Google Scholar
  22. Crowne, D. D., and Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting and Clinical Psychology, 24, 349–354.CrossRefGoogle Scholar
  23. Eyberg, S., and Robinson, E. (1983). Dyadic Parent Child Interaction Coding System: A manual. Psychological Documents, 13. ( Ms. No. 2582 )Google Scholar
  24. Fenigstein, A., Scheier, M. F., and Buss, A. H. (1975). Public and private self-consciousness: Assessment and theory. Journal of Consulting and Clinical Psychology, 43, 522–527.CrossRefGoogle Scholar
  25. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378–382.CrossRefGoogle Scholar
  26. Fleiss, J. I.., Nee, J. C. M., and Landis, J. R. (1979). The large sample variance of kappa in the case of different sets of raters. Psychological Bulletin, 86, 974–977.CrossRefGoogle Scholar
  27. Floyd, F. J., and Markman, H. J. (1983). Observational biases in spouse observation: Toward a cognitive/behavioral model of marriage. Journal of Consulting and Clinical Psychology, 51, 450–457.Google Scholar
  28. Glass, G. V., and Stanley, J. C. (1970). Statistical methods in education and psychology ( 2nd ed. ). Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
  29. Goldfried, M., and Kent, R. (1972). Traditional versus behavioral personality assessment: A comparison of methodological and theoretical assumptions. Psychological Bulletin, 77, 409–420.PubMedCrossRefGoogle Scholar
  30. Gottman, J. M. (1980). Analyzing for sequential connection and assessing interobserver reliability for the sequential analysis of observational data. Behavioral Assessment, 2, 361–368.Google Scholar
  31. Hadley, T., and Jacob, T. (1973). Relationship among four measures of family power. Journal of Personality and Social Psychology, 27, 6–12.CrossRefGoogle Scholar
  32. Hadley, T. R., and Jacob, T. (1976). The measurement of family power: A methodological study. Sociometry, 39, 384–395.CrossRefGoogle Scholar
  33. Haley, J. (1967). Cross-cultural experimentation: An initial attempt. Human Organization, 26, 110–117.Google Scholar
  34. Hannum, J. W., and Mayer, J. M. (1984). Validation of two family assessment approaches. journal of Marriage and Family, 46, 741–748.CrossRefGoogle Scholar
  35. Harris, A. (1969). Observer effect on family interaction. Unpublished doctoral dissertation, University of Oregon.Google Scholar
  36. Harris, F. C., and Lahey, B. B. (1982). Subject reactivity in direct observational assessment: A review and critical analysis. Clinical Psychology Review, 2, 523–538.CrossRefGoogle Scholar
  37. Hartmann, D. P. (1977). Considerations in the choice of interobserver reliability estimates. journal of Applied Behavior Analysis, 10, 103–116.PubMedCrossRefGoogle Scholar
  38. Haynes, S. N., and Horn, W. F. (1982). Reactivity in behavioral observation: A review. Behavioral Assessment, 4, 369–385.Google Scholar
  39. Haynes, S. N., Chavez, R. E., and Samuel, V. (1984). Assessment of marital communication distress. Behavioral Assessment, 6, 315–321.Google Scholar
  40. Henggeler, S. W., Borduin, C. M., Rodick, J. D., and Tavormina, J. (1979). Importance of task content for family interaction research. Developmental Psychology, 15, 660–661.CrossRefGoogle Scholar
  41. Herbert, J., and Attridge, C. (1975). A guide for developers and users of observation systems and manuals. The American Education Research journal, 12, 1–20.CrossRefGoogle Scholar
  42. Hollenbeck, A. R. (1978). Problems of reliability on observational research. In G. Sackett (Ed.), Observing behavior: Vol. 2. Data collection and analysis methods. Baltimore: University Park Press.Google Scholar
  43. Hoover, L. K., and Rinehart, H. H. (1968). The effect of an outside observer on family interaction. Unpublished manuscript.Google Scholar
  44. Hughes, H. M., and Haynes, S. N. (1978). Structured laboratory observation in the behavioral assessment of parent-child interactions: A methodological critique. Behavior Therapy, 9, 428–447.Google Scholar
  45. Jacob, T. (1975). Family interaction in normal and disturbed families: A methodological and substantive review. Psychological Bulletin, 82, 33–65.PubMedCrossRefGoogle Scholar
  46. Jacob, T. (1976). Behavioral assessment of marital dysfunction. In M. Hersen and A. Bellack (Eds.), Behavioral assessment: A practical handbook. New York: Pergamon Press.Google Scholar
  47. Jacob, T., and Davis, J. (1973). Family interaction as a function of experimental task. Family Process, 12, 415–427.CrossRefGoogle Scholar
  48. Jacob, T., Grounds, L., and Haley, R. (1982). Correspondence between parents’ reports on the Behavior Problem Checklist. journal of Abnormal Child Psychology, 4, 593–608.CrossRefGoogle Scholar
  49. Jacob, T., Rushe, R. H., and Tennenbaum, D. L. (1986). Alcoholism and family interaction: An experimental paradigm. Unpublished manuscript, University of Pittsburgh.Google Scholar
  50. Johnson, S. M., and Bolstad, O. D. (1973). Methodological issues in naturalistic observation: Some problems and solutions for field research. In L. A. Hamerlynck, L. C. Handy, and E. J. Mash (Eds.), Behavior change: Methodology, concepts and practice. Champaign, IL: Research Press.Google Scholar
  51. Johnson, S. M., and Bolstad, O. D. (1975). Reactivity to home observation: A comparison of audio recorded behavior with observers present or absent. Journal of Applied Behavioral Analysis, 8, 181–185.CrossRefGoogle Scholar
  52. Johnson, S. M., Christensen, A., and Bellamy, G. T. (1976). Evaluation of family intervention through unobtrusive audio recordings: Experiences in “bugging” children. journal of Applied Behavior Analysis, 9, 213–219.Google Scholar
  53. Jones, R. R., Reid, J. B., and Patterson, G. R. (1975). Naturalistic observation in clinical assessment. In P. McReynolds (Ed.), Advances in Psychological Assessment (Vol. 3). San Francisco: Jossey-Bass.Google Scholar
  54. Kazdin, A. E. (1977). Artifact, bias, and complexity of assessment: The ABC’s of reliability. Journal of Applied Behavior Analysis, 10, 141–150.PubMedCrossRefGoogle Scholar
  55. Kazdin, A. E. (1982). Observer effects: Reactivity of direct observation. In D. P. Hartman (Ed.), Using observers to study behavior. San Francisco: Jossey-Bass.Google Scholar
  56. Kent, R. N., Kanowitz, J., O’Leary, K. D., and Cheiken, M. (1977). Observer reliability as a function of circumstances of assessment. journal of Applied Behavioral Analysis, 10, 317–324.CrossRefGoogle Scholar
  57. Krahn, G. L., and Gabriel, R. M. (1984). Quantifying categorical observations of social interactions through multidimensional scaling. Developmental Psychology, 20, 833–843.Google Scholar
  58. Lytton, H. (1974). Comparative yield of three data sources in the study of parent-child interaction. Merrill-Palmer Quarterly, 20, 53–64.Google Scholar
  59. Lytton, H. (1979). Disciplinary encounters between young boys and their mothers and fathers: Is there a contingency system? Developmental Psychology, 15, 256–268.CrossRefGoogle Scholar
  60. Margolin, G. (1978). Relationships among marital assessment procedures: A correlational study. Journal of Consulting and Clinical Psychology, 46, 1556–1558.CrossRefGoogle Scholar
  61. Margolin, G., and Weiss, R. L. (1978). Comparative evaluation of therapeutic components associated with behavioral marital treatments. Journal of Consulting and Clinical Psychology, 46, 1476–1486.PubMedCrossRefGoogle Scholar
  62. Mash, E. J., and Johnston, C. (1982). A comparison of the mother-child interactions of younger and older hyperactive and normal children. Child Development, 53, 1371–1381.PubMedCrossRefGoogle Scholar
  63. Mezzich, J. E., Kraemer, H. C., Worthington, D. R. L., and Coffman, G. A. (1981). Assessment of agreement among several raters formulating multiple diagnoses. Journal of Psychiatric Research, 16, 29–39.PubMedCrossRefGoogle Scholar
  64. Mitchell, S. K. (1979). Interobserver agreement, reliability and generalizability of data collected in observational studies. Psychological Bulletin, 86(2), 376–390.Google Scholar
  65. Moos, R., and Moos, B. S. (1981). Family Environment Scale: Manual. Palo Alto: Consulting Psychologists Press.Google Scholar
  66. Murrell, S. A. (1971). Family interaction variables and adjustment of nonclinic boys. Child Development, 42, 1485–1494.CrossRefGoogle Scholar
  67. Nunnally, J. C. (1967). Psychometric theory. New York: McGraw-Hill.Google Scholar
  68. Oliven, M. E., and Reiss, D. (1984). Family concepts and their measurement: Things are seldom what they seem. Family Process, 23, 33–48.CrossRefGoogle Scholar
  69. Olson, D. H. (1985). Commentary: Struggling with congruence across theoretical models and methods. Family Process, 24, 203–207.CrossRefGoogle Scholar
  70. Olson, D. H., and Portner, J. (1983). Family adaptability and cohesion evaluation scales. In E. E. Filsinger (Ed.), Marriage and family assessment. Beverly Hills, CA: Sage Publications.Google Scholar
  71. Olson, D. H., and Ryder, R. G. (1970). Inventory of Marital Conflicts (IMC): An experimental interaction procedure. Journal of Marriage and the Family, 32, 443–448.CrossRefGoogle Scholar
  72. O’Neill, M. S., and Alexander, J. F. (1970, April). Family interaction as a function of task characteristics. Paper presented at Rocky Mountain Psychological Association, Salt Lake City, UT.Google Scholar
  73. O’Rourke, V. (1963). Field and laboratory: The decision-making behavior of family groups in two experimental conditions. Sociometry, 26, 422–435.CrossRefGoogle Scholar
  74. Patterson, G. R. (1982). A social learning approach: Vol. 3. Coercive family process. Eugene, OR: Castalia.Google Scholar
  75. Patterson, G. R., and Cobb, J. A. (1971). A dyadic analysis of “aggressive” behavior. In J. P. Hill (Ed.), Minnesota symposia on child psychology (Vol. 5 ). Minneapolis: University of Minnesota Press.Google Scholar
  76. Patterson, G. R., and Moore, D. (1979). Interactive patterns as units of behavior. In M. E. Lamb, S. J. Suomi, Sc G. R. Stephenson (Ed.), Social interaction analysis: Methodological issues. Madison: University of Wiscons in Press.Google Scholar
  77. Patterson, G. R., Sc Reid, J. B. (1970). Reciprocity and coercion: Two facets of social systems. In C. Neuringer and J. D. Michael (Eds.), Behavior modification in clinical psychology. New York: AppletonCentury-Crofts.Google Scholar
  78. Roberts, R., Jr., and Renzaglia, A. (1965). The influence of tape recording on counseling. Journal of Counseling Psychology, 12, 10–16.Google Scholar
  79. Robinson, E. A., and Price, M. G. (1980). Pleasurable behavior in marital interaction: An observation study. Journal of Consulting and Clinical Psychology, 48, 117–118.Google Scholar
  80. Rosenberg, M. J. (1969). The conditions and consequences of evaluation apprehension. In R. Rosenthal and R. L. Roshow (Eds.), Artifact in behavioral research. New York: Academic Press.Google Scholar
  81. Rosenthal, R. (1966). Experimenter effects in behavioral research. New York: Appleton-Century.Google Scholar
  82. Ross, G., Kagan, J., Zelazo, P., and Kotelchuck, M. (1975). Separation protest in infants in home and laboratory. Developmental Psychology, 11, 256–257.CrossRefGoogle Scholar
  83. Scott, W., and Wertheimer, M. (1962). Introduction to psychological research. New York: Wiley.Google Scholar
  84. Sigafoos, A., Reiss, D., Rich, J., and Douglas, E. (1985). Pragmatics in the measurement of family functioning: An interpretive framework for methodology. Family Process, 24, 189–203.PubMedCrossRefGoogle Scholar
  85. Skinner, H., Steinhauer, P., and Santa-Barbara, J. (1983). Family Assessment Measure. Toronto, Ontario: Addiction Research Foundation.Google Scholar
  86. Skinner, H. A., Steinhauer, P. D., and Santa-Barbara, J. (1983). The Family Assessment Measure. Canadian Journal of Community Mental Health, 3 (2), 91–104.Google Scholar
  87. Stein, S.J., Girodo, M., and Dotzenroth, S. (1982). The interrelationships and reliability of a multilevel behavior-based assessment package for distressed couples. Journal of Behavioral Assessment, 4, 343–360.Google Scholar
  88. Steinglass, P. (1980). Assessing families in their own homes. American journal of Psychiatry, 137, 1523–1529.PubMedGoogle Scholar
  89. Susman, E. J., Peters, D. J., and Steward, R. (1976). Naturalistic observational child study: A review. Paper presented at the 4th Biannual Southeastern Conferences on Human Development, Nashville.Google Scholar
  90. Tennenbaum, D. L. (1980). The effect of observer salience on family interaction in the home. Unpublished master’s thesis, University of Pittsburgh.Google Scholar
  91. Tennenbaum, D. L., and Jacob, T. ( 1985, November). An investigation of reactivity effects in spouse observation. Poster presented at Association for the Advancement of Behavior Therapy ( AABT) Convention, Houston.Google Scholar
  92. Tronick, E., Als, H., and Brazelton, T. B. (1977). Mutuality in mother-infant interaction. journal of Communication, 7, 74–79.CrossRefGoogle Scholar
  93. Vidich, A. J. (1956). Methodological problems in the observation of husband-wife interactions. Marriage and Family Living, 18, 234–239.CrossRefGoogle Scholar
  94. Vincent, J. P., Friedman, L. C., Nugent, J., and Messerly, L. (1979). Demand characteristics in observations of marital interaction. Journal of Consulting and Clinical Psychology, 47, 557–566.PubMedCrossRefGoogle Scholar
  95. Volkin, J. I., and Jacob, T. (1981). The impact of spouse monitoring on target behavior and recorder satisfaction. Journal of Behavioral Assessment, 3, 99–109.CrossRefGoogle Scholar
  96. Watzlawick, P. (1966). A structured family interview. Family Process, 5, 256–271.CrossRefGoogle Scholar
  97. Weber, S. J., and Cook, T. D. (1972). Subject effects in laboratory research: An examination of subject roles, demand characteristics, and valid interference. Psychological Bulletin, 77, 273–295.CrossRefGoogle Scholar
  98. Weinrott, M. R., Garrett, B., and Todd, N. (1978). The influence of observer presence on classroom behavior. Behavior Therapy, 9, 900–911.CrossRefGoogle Scholar
  99. Weiss, R. L., and Perry, B. A. (1983). The Spouse Observation Checklist: Development and clinical application. In E. E. Filsinger (Ed.), Marriage and family assessment. Beverly Hills, CA: Sage Publications.Google Scholar
  100. Weiss, R. L., and Summers, K. J. (1983). Marital Interaction Coding System-Ill. In E. Filsinger (Ed.), Marriage and family assessment. Beverly Hills, CA: Sage Publications.Google Scholar
  101. Wells, K. C., McMahon, R. J., Forehand, R., and Griest, D. L. (1980). Effect of a reliability observer on the frequency of positive parent behavior recorded during naturalistic parent-child interactions. Journal of Behavioral Assessment, 2, 65–69.CrossRefGoogle Scholar
  102. White, G. D. (1973). Effects of observer presence on mother and child behavior. Unpublished doctoral dissertation, University of Oregon, Eugene.Google Scholar
  103. White, G. D. (1977). The effects of observer presence on the activity level of families. Journal of Applied Behavior Analysis, 10, 734.CrossRefGoogle Scholar
  104. Wieder, G. B., and Weiss, R. L. (1980). Generalizability theory and the coding of marital interactions. Journal of Consulting and Clinical Psychology, 48, 469–477.PubMedCrossRefGoogle Scholar
  105. Wiggins, J. S. (1973). Observational techniques: 1. Generalizability and facets of observation. Personality and prediction principles of personality assessment. Reading, MA: Addison-Wesley.Google Scholar
  106. Yarrow, M. R., and Waxier, C. Z. (1979). Observing interaction: A confrontation with methodology. In R. B. Cairns, (Ed.), The analysis of social interaction: Methods, issues, and illustrations. New York: Erlbaum.Google Scholar
  107. Zajonc, R. B. (1965). Social facilitation. Science, 149, 269–274.PubMedCrossRefGoogle Scholar
  108. Zegiob, L. E., and Forehand, R. (1975). Maternal interactive behavior as a function of socioeconomic status, race and sex of child. Child Development, 46, 564–568.CrossRefGoogle Scholar
  109. Zegiob, L. E., and Forehand, R. (1978). Parent-child interactions: Observer effects and social class differences. Behavior Therapy, 9, 118–123.CrossRefGoogle Scholar
  110. Zegiob, L. E., Arnold, S., and Forehand, R. (1975). An examination of observer effects in parent-child interactions. Child Development, 46, 507–512.Google Scholar
  111. Zuckerman, E., and Jacob, T. (1979). Task effects in family interaction. Family Process, 18, 47–53.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 1987

Authors and Affiliations

  • Theodore Jacob
    • 1
  • Daniel L. Tennenbaum
    • 2
  • Gloria Krahn
    • 3
  1. 1.Division of Family Studies, 210 Family and Consumer Resources BuildingUniversity of ArizonaTusconUSA
  2. 2.Department of PsychiatryUniversity of PittsburghPittsburghUSA
  3. 3.Crippled Children’s DivisionOregon Health Sciences UniversityPortlandUSA

Personalised recommendations