How the Granularity of Evaluation Affects Reliability of Peer-Assessment Modelization in the OpenAnswer System

  • Maria De Marsico
  • Andrea Sterbini
  • Marco Temperini
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8534)


The OpenAnswer system has the goal of exploiting teacher mediated peer-assessment for the evaluation of answers to open ended questions. The system models both the learning state of each student and their choices during peer-assessment. In OpenAnswer, each student is represented as a Bayesian network made of a triple of finite-domain variables: K for student’s Knowledge about a topic, J for the estimated ability to evaluate (”Judge”) the answer of another peer, C for Correctness of the answer to a given question. The student’s individual sub-networks are connected through further Bayesian variables which model each peer-assessment choice, depending on the type of peer-assessment performed: (G for grading, B for choosing the best, W for choosing the worst). During an assessment session, each student grades a fixed number of peers’ answers. The final result for a given session is a full set of grades for all students’ answers, although the teacher had actually graded only a part of them. The student’s assessments are instantiated in the network as evidence, together with the teacher’s (perhaps partially complete) grades, so that OpenAnswer deduces the remaining grades. In the former OpenAnswer implementation, all variables were represented through a probability distribution over three values (Good/Fair/Bad for K and J, correct/fair/wrong for C). We present experiments and simulations showing that, by increasing the domain granularity for all variables from 3 to 6 values (A to F), the information obtained from the Bayesian network achieves higher reliability.


assessment peer-assessment social collaborative e-learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Limongelli, C., Sciarrone, F., Temperini, M., Vaste, G.: Lecomps5: A web-based learning system for course personalization and adaptation. In: Nunes, M.B., McPherson, M. (eds.) IADIS International Conference e-Learning 2008, Amsterdam, The Netherlands, July 22-25, vol. 1, pp. 325–332 (2008)Google Scholar
  2. 2.
    Popescu, E.: Adaptation Provisioning with respect to Learning Styles in a Web-Based Educational System: An Experimental Study. Journal of Computer Assisted Learning 26(4), 243–257 (2010)Google Scholar
  3. 3.
    Sterbini, A., Temperini, M.: Selection and sequencing constraints for personalized courses. In: Proc. 40th IEEE Frontiers in Education (FIE) Conference, Washington, DC, USA, October 27-30, pp. T2C1–T2C6 (2010), doi:10.1109/FIE.2010.5673146Google Scholar
  4. 4.
    Essalmi, F., Jemni Ben Ayed, L., Jemni, M., Kinshuk, Graf, S.: A fully Personalization Strategy of E-Learning Scenarios. Computers in Human Behavior 26(4), 581–591 (2010)Google Scholar
  5. 5.
    Limongelli, C., Sciarrone, F., Temperini, M., Vaste, G.: The Lecomps5 Framework for Personalized Web-Based Learning: a Teacher’s Satisfaction Perspective. Computers in Human Behavior 27(4), 1310–1320 (2011) ISSN 0747-5632Google Scholar
  6. 6.
    Kirschner, P.A.: Using integrated electronic environments for collaborative teaching/learning. Learning and Instruction 10, 1–9 (2001)CrossRefGoogle Scholar
  7. 7.
    Sterbini, A., Temperini, M.: Learning from Peers: Motivating Students through Reputation Systems. In: Proc. IEEE/IPSJ International Symposium on Applications and the Internet, July 28-August 01, pp. 305–308 (2008), Scholar
  8. 8.
    Kreijns, K., Kirschner, P.A., Jochems, W.: Identifying the pitfalls for social interaction in computer supported collaborative learning environments: a review of the research. Computers in Human Behavior 19, 335–353 (2003)Google Scholar
  9. 9.
    Cheng, Y., Ku, H.: An investigation of the effects of reciprocal peer tutoring. Computers in Human Behavior 25, 40–49 (2009)Google Scholar
  10. 10.
    Popescu, E.: Providing Collaborative Learning Support with Social Media in an Integrated Environment. World Wide Web (2012), doi:10.1007/s11280-012-0172-6, ISSN: 1386-145XGoogle Scholar
  11. 11.
    De Marsico, M., Sterbini, A., Temperini, M.: The Definition of a Tunneling Strategy between Adaptive Learning and Reputation-based Group Activities. In: Proc. 11th IEEE International Conference on Advanced Learning Technologies, ICALT 2011, Athens, GA, USA, July 6-8, pp. 498–500 (2011), doi:10.1109/ICALT.2011.155Google Scholar
  12. 12.
    De Marsico, M., Sterbini, A., Temperini, M.: A strategy to join adaptive and reputation-based social-collaborative e-learning, through the Zone of Proximal Development. Int. Journal of Distance Education Technology, IJDET 11(3), 12–31 (2013)Google Scholar
  13. 13.
    De Marsico, M., Sterbini, A., Temperini, M.: A Framework to Support Social-Collaborative Personalized e-Learning. In: Kurosu, M. (ed.) Human-Computer Interaction, HCII 2013, Part II. LNCS, vol. 8005, pp. 351–360. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  14. 14.
    Palmer, R.: On-line assessment and free-response input – a pedagogic and technical model for squaring the circle. In: Proceedings of the 7th CAA Conference, Loughborough University (2003)Google Scholar
  15. 15.
    Sterbini, A., Temperini, M.: Correcting open-answer questionnaires through a Bayesian-network model of peer-based assessment. In: Proc. Int. Conference on Information Technology Based Higher Education and Training, ITHET 2012, pp. 1–6 (2012)Google Scholar
  16. 16.
    Sterbini, A., Temperini, M.: OpenAnswer, a framework to support teacher’s management of open answers through peer assessment. In: Proc. 43th ASEE/IEEE Frontiers in Education, FIE, Oklahoma City, OK, October 23-26. IEEE Computer Society (2013)Google Scholar
  17. 17.
    Sterbini, A., Temperini, M.: Analysis of OpenAnswers via mediated peer-assessment. In: Proc. 17th IEEE Int Conf. on System Theory, Control and Computing, ICSTCC 2013 (2013)Google Scholar
  18. 18.
    Yamanishi, K., Li, H.: Mining Open Answers in Questionnaire Data. IEEE Intelligent Systems, 58–63 (2002)Google Scholar
  19. 19.
    Morinaga, S., Yamanishi, K., Tateishi, K., Fukushima, T.: Mining product reputations on the Web. In: Proc. 8th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, KDD 2002, pp. 341–349 (2002)Google Scholar
  20. 20.
    Jackson, K., Trochim, W.: Concept mapping as an alternative approach for the analysis of open-ended survey responses. Organizational Research Methods 5, 307–336 (2002)Google Scholar
  21. 21.
    Romero, C., Ventura, S.: Educational Data Mining: A Review of the State of the Art. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 40(6), 601–618 (2010)Google Scholar
  22. 22.
    Castellanos-Nieves, D., Fernández-Breis, J., Valencia-García, R., Martínez-Béjar, R., Iniesta-Moreno, M.: Semantic Web Technologies for supporting learning assessment. Information Sciences 181(9), 1517–1537 (2011)Google Scholar
  23. 23.
    El-Kechaï, N., Delozanne, É., Prévit, D., Grugeon, B., Chenevotot, F.: Evaluating the Performance of a Diagnosis System in School Algebra. In: Leung, H., Popescu, E., Cao, Y., Lau, R.W.H., Nejdl, W. (eds.) ICWL 2011. LNCS, vol. 7048, pp. 263–272. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  24. 24.
    Formisano, A., Omodeo, E.G., Temperini, M.: Layered map reasoning: An experimental approach put to trial on sets. Electronic Notes in Theoretical Computer Science 48, 1–28 (2001)Google Scholar
  25. 25.
    Chung, H., Graf, S., Robert Lai, K., Kinshuk: Enrichment of Peer Assessment with Agent Negotiation. IEEE Trans. on Learning Technologies (TLT) 4(1), 35–46 (2011)Google Scholar
  26. 26.
    Sterbini, A., Temperini, M.: Supporting Assessment of Open Answers in a Didactic Setting. In: Proc. 12th IEEE Int. Conference on Advanced Learning Technologies (ICALT 2012), Rome, Italy, July 4-6 (2012)Google Scholar
  27. 27.
    Sterbini, A., Temperini, M.: Dealing with open-answer questions in a peer-assessment environment. In: Popescu, E., Li, Q., Klamma, R., Leung, H., Specht, M. (eds.) ICWL 2012. LNCS, vol. 7558, pp. 240–248. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  28. 28.
    Costa Santos, V.: CLP (BN): Constraint logic programming for probabilistic knowledge. In: Proc. 19th Conf. on Uncertainty in Artificial Intelligence. Morgan Kaufmann (2002)Google Scholar
  29. 29.
    Moij, J.M.: liDAI: a free and open source C++ library for Discrete Approximate Inference in graphical models. J. of Machine Learning Research 11, 2169–2173 (2010)Google Scholar
  30. 30.
    Kschischang, F.R., Frey, B.J., Loeliger, H.-A.: Factor graphs and the sum-product algorithm. IEEE Trans. on Information Theory 47(2), 498–519 (2001)Google Scholar
  31. 31.
    Bloom, B.S., Mesia, B.B., Krathwohl, D.R.: Taxonomy of Educational Objectives (two vols: The Affective Domain & The Cognitive Domain), NY. David McKay (1964)Google Scholar
  32. 32.
    Chatti, M.A., Sodhi, T., Specht, M., Klamma, R., Klemke, R.: u-Annotate: An application for user-driven freeform digital ink annotation of e-Learning content. In: Proc. Sixth International Conference on Advanced Learning Technologies, ICALT 2006, pp. 1039–1043 (2006)Google Scholar
  33. 33.
    Limongelli, C., Sciarrone, F., Starace, P., Temperini, M.: An Ontology-driven OLAP System to Help Teachers in the Analysis of Web Learning Object Repositories. Information Systems Management 27(3), 198–206, doi:10.1080/10580530.2010.493810Google Scholar
  34. 34.
    Dyckhoff, A.L., Zielke, D., Bültmann, M., Chatti, M.A., Schroeder, U.: Design and implementation of a learning analytics toolkit for teachers. Educational Technology and Society 15(3), 58–76 (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Maria De Marsico
    • 1
  • Andrea Sterbini
    • 1
  • Marco Temperini
    • 2
  1. 1.Dept. of Computer ScienceSapienza University of RomeItaly
  2. 2.Dept. of Computer, Control, and Management EngineeringSapienza University of RomaItaly

Personalised recommendations