Peer observations among faculty in a college of education: investigating the summative and formative uses of the Reformed Teaching Observation Protocol (RTOP)

  • Audrey Amrein-Beardsley
  • Sharon E. Osborn Popp


Teacher educators piloted the use of the Reformed Teaching Observation Protocol (RTOP), a peer observation instrument associated with increases in learning in science and mathematics teacher education courses. Faculty participants received a series of trainings in RTOP use and rated each other’s teaching during multiple peer observations. The purpose of this study was to investigate whether the RTOP would prove useful for formative and summative purposes across teacher education courses in general. While participants saw value in the peer observation process and the RTOP instrument, findings suggest that the perceived formative functions of the RTOP outweighed the instrument’s summative value.


Higher-education Instructional method Many-faceted Rasch model Peer observation Professional development 


  1. Adamson, S. L., Banks, D., Burtch, M., Cox, F., Judson, E., Turley, J. B., et al. (2003). Reformed undergraduate instruction and its subsequent impact on secondary school teaching practice and student achievement. Journal of Research in Science Teaching, 40(10), 939–957. doi: 10.1002/tea.10117.CrossRefGoogle Scholar
  2. Andrich, D. (1978). A rating formulation for ordered categories. Psychometrika, 43, 357–374.CrossRefGoogle Scholar
  3. Bernstein, D. J. (2008). Peer review and evaluation of the intellectual work of teaching. Change: The Magazine of Higher Learning, 40(2), 48–51. doi: 10.3200/CHNG.40.2.48-51.CrossRefGoogle Scholar
  4. Bingham, R., & Ottewill, R. (2001). Whatever happened to peer review? Revitalising the contribution of tutors to course evaluation. Quality Assurance in Education, 9(1), 22–39. doi: 10.1108/09684880110381319.CrossRefGoogle Scholar
  5. Blackmore, J. A. (2005). A critical evaluation of peer review via teaching observation within higher education. International Journal of Educational Management, 19(3), 218–232. doi: 10.1108/09513540510591002.CrossRefGoogle Scholar
  6. Boyer, E. L. (1987). College: The undergraduate years. New York: Harper and Row Publishers Inc.Google Scholar
  7. Cole, R. E. (2003). New organizational designs for sustainable quality improvement. Keynote address delivered at the 6th International Conference on Quality Management and Organisational Development (QMOD), Paris, France.Google Scholar
  8. Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., & Hanson, W. E. (2003). Advanced mixed method research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 209–240). Thousand Oaks: Sage.Google Scholar
  9. Cross, K. P. (1986). A proposal to improve teaching – or – what taking teaching seriously should mean. AAHE Bulletin, 39(1), 9–14.Google Scholar
  10. Dommeyer, C. J., Baum, P., Hanna, R., & Chapman, K. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29(5), 611–623. doi: 10.1080/02602930410001689171.CrossRefGoogle Scholar
  11. Erickson, F. (1986). Qualitative methods in research on teaching. In M. Wittrock (Ed.), Handbook of Research on Teaching, vol 3 (pp. 119–161). New York: Macmillan.Google Scholar
  12. Gijbels, D., van de Watering, G., Dochy, F., & van den Bossche, P. (2006). New learning environments and constructivism: The students’ perspective. Instructional Science: An International Journal of Learning and Cognition, 34(3), 213–226. doi: 10.1007/s11251-005-3347-z.Google Scholar
  13. Gosling, D. (2002). Models of peer observation of teaching. Birmingham: Learning and Teaching Support Network Generic Centre Learning and Teaching Support Network.Google Scholar
  14. Greene, J. C. (2007). Mixed methods social inquiry. San Francisco: Jossey-Bass.Google Scholar
  15. Hammersley-Fletcher, L., & Orsmond, P. (2005). Reflecting on reflective practices within peer observation. Studies in Higher Education, 30(2), 213–224. doi: 10.1080/03075070500043358.CrossRefGoogle Scholar
  16. Heitzmann, R. (2010). Ten suggestions for enhancing lecturing. Education Digest, 75(9), 50–54.Google Scholar
  17. Hendry, G. D., Cumming, R. G., Lyon, P. M., & Gordon, J. (2001). Student-centered course evaluation in a four-year, problem based medical programme: Issues in collection and management of feedback. Assessment & Evaluation in Higher Education, 26(4), 327–339.CrossRefGoogle Scholar
  18. Hutchings, P. (1995). From idea to prototype: The peer review of teaching. Sterling: Stylus.Google Scholar
  19. Hutchings, P. (1996). Making teaching community property: A menu for peer collaboration and peer review. Sterling: Stylus.Google Scholar
  20. Hutchings, P. (1998). The course portfolio. Sterling: Stylus.Google Scholar
  21. Keig, L. (2000). Formative peer review of teaching: attitudes of faculty at liberal arts colleges towards colleague assessment. Journal of Personnel Evaluation in Education, 14(1), 67–87.CrossRefGoogle Scholar
  22. Lawson, A. E. (2003). Using the RTOP to evaluate reformed science and mathematics instruction: Improving undergraduate instruction in science, technology, engineering, and mathematics. Committee on Undergraduate Science Education. Washington D.C.: National Academies Press, National Research Council.Google Scholar
  23. Lawson, A. E., Benford, R., Bloom, I., Carlson, M. P., Falconer, K., Hestenes, D., et al. (2002). Evaluating college science and mathematics instruction. Journal of College Science Teaching, 36, 388–393.Google Scholar
  24. Linacre, J. M. (2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3(1), 85–106.Google Scholar
  25. Linacre, J. M. (2009). A user’s guide to FACETS. Chicago: Scholar
  26. Linacre, J. M., & Wright, B. D. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8, 370. Retrieved from
  27. Linacre, J. M., Engelhard, G., Jr., Tatum, D. S., & Myford, C. M. (1994). Measurement with judges: many-faceted conjoint measurement. International Journal of Educational Research, 21(6), 569–577. doi: 10.1016/0883-0355(94)90011-6.CrossRefGoogle Scholar
  28. Loyens, S. M. M., Rikers, R. M. J. P., & Schmidt, H. G. (2009). Students’ conceptions of constructivist learning in different programme years and different learning environments. British Journal of Educational Psychology, 79(3), 501–514. doi: 10.1348/000709908X378117.CrossRefGoogle Scholar
  29. MacIsaac, D., Sawado, D., & Falconer, K. (2001). Using the Reformed Teaching Observation Protocol (RTOP) as a catalyst for self-reflective change in secondary science teaching. Paper presented at annual meeting of the American Educational Research Association, Seattle, WA.Google Scholar
  30. Marsh, H. W., Muthen, B., Asparouhov, T., Ludtke, O., Robitzsch, A., Morin, A. J. S., et al. (2009). Exploratory structural equation modeling, integrating CFA and EFA: Application to students’ evaluations of university teaching. Structural Equation Modeling: A Multidisciplinary Journal, 16(3), 439–476. doi: 10.1037/a0019227.CrossRefGoogle Scholar
  31. Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174. doi: 10.1007/BF02296272.CrossRefGoogle Scholar
  32. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks: Sage Publications.Google Scholar
  33. Ngware, M. W. (2005). An improvement in instructional quality: can evaluation of teaching effectiveness make a difference? Quality Assurance in Education, 13(3), 183–201. doi: 10.1108/09684880510607936.CrossRefGoogle Scholar
  34. Peel, D. (2005). Peer observation as a transformatory tool? Teaching in Higher Education, 10(4), 489–504. doi: 10.1080/13562510500239125.CrossRefGoogle Scholar
  35. Piburn, M., Sawada, D., Turley, J., Falconer, K., Benford, R., Bloom, I., et al. (2000). Reformed Teaching Observation Protocol (RTOP) reference manual. ACEPT Technical Report No. IN00-3. Tempe: Arizona Board of Regents.Google Scholar
  36. Rasch, G. (1960/1980). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Educational Research, 1960. Expanded edition, Chicago: The University of Chicago Press, 1980.Google Scholar
  37. Roberts, T. G., Irani, T. A., Telg, R. W., & Lundy, L. K. (2005). The development of an instrument to evaluate distance education courses using student attitudes. American Journal of Distance Education, 19(1), 51–64. doi: 10.1207/s15389286ajde1901_5.CrossRefGoogle Scholar
  38. Roche, L. A., & Marsh, H. W. (2000). Multiple dimensions of university teacher self-concept: construct validation and the influence of students’ evaluations of teaching. Instructional Science, 28(5–6), 439–468.CrossRefGoogle Scholar
  39. Shulman, L. S. (1999). Taking learning seriously. Change, 31(4), 10–17. doi: 10.1080/00091389909602695.CrossRefGoogle Scholar
  40. Swinglehurst, D., Russell, J., & Greenhalgh, T. (2008). Peer observation of teaching in the online environment: an action research approach. Journal of Computer Assisted Learning, 24(5), 383–393. doi: 10.1111/j.1365-2729.2007.00274.x.CrossRefGoogle Scholar
  41. Tinto, V. (1997). Classrooms as communities: exploring the educational character of student persistence. Journal of Higher Education, 68(6), 599–623. doi: 10.2307/2959965.CrossRefGoogle Scholar
  42. Wilson, M. (2005). Constructing measures: an item response modeling approach. Mahwah: Lawrence Erlbaum Associates.Google Scholar
  43. Wood, L. N., & Harding, A. (2007). Can you show you are a good lecturer? International Journal of Mathematical Education in Science and Technology, 38(7), 939–947.CrossRefGoogle Scholar
  44. Yon, M., Burnap, C., & Kohut, G. (2002). Evidence of effective teaching: Perceptions of peer reviewers. College Teaching, 50(3), 104–110. doi: 10.1080/87567550209595887.CrossRefGoogle Scholar

Copyright information

© Springer Science + Business Media, LLC 2011

Authors and Affiliations

  • Audrey Amrein-Beardsley
    • 1
  • Sharon E. Osborn Popp
    • 2
  1. 1.Mary Lou Fulton Teachers CollegeArizona State UniversityPhoenixUSA
  2. 2.Western Regional Examining BoardPhoenixUSA

Personalised recommendations