Abstract
This paper presents a study describing the development of an Evaluation Framework (EF) for data competitions in TEL. The study applies the Group Concept Method (GCM) to empirically depict criteria and their indicators for evaluating software applications in TEL. A statistical analysis including multidimensional scaling and hierarchical clustering on the GCM data identified the following six evaluation criteria: 1.Educational Innovation, 2.Usability, 3.Data, 4.Performance, 5.Privacy, and 6.Audience. Each of them was operationalized through a set of indicators. The resulting Evaluation Framework (EF) incorporating these criteria was applied to the first data competition of the LinkedUp project. The EF was consequently improved using the results from reviewers’ interviews, which were analysed qualitatively and quantitatively. The outcome of these efforts is a comprehensive EF that can be used for TEL data competitions and for the evaluation of TEL tools in general.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Drachsler, H., Verbert, K., Manouselis, N., Vuorikari, R., Wolpers, M., Lindstaedt, S.: Preface (Special issue on dataTEL - Data Supported Research in Technology-Enhanced Learning). Int. J. Technol. Enhanc. Learn. 4(1/2) (2012)
d’Aquin, M., Dietze, S., Herder, E., Drachsler, H.: Using linked data in learning analytics. eLearning Papers. Nr. 37 (2014), http://www.openeducationeuropa.eu/en/elearning_papers , ISSN: 1887-1542
Benson, L., Elliot, D., Grant, M., Holschuh, D., Kim, B., Kim, H., Lauber, E., Loh, S., Reeves, T.: Usability and Instructional Design Heuristics for E-Learning Evaluation. In: Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications, Denver, Colorado, USA, pp. 1615–1621 (2002)
Granic, A., Cukusic, M.: Usability Testing and Expert Inspections Complemented by Educational Evaluation: A Case Study of an e-Learning Platform. Educational Technology & Societ 14(2), 107–123 (2011)
Sun, P.-C., Tsai, R.J., Finger, G., Chen, Y.-Y., Yeh, D.: What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers and Education 50(4), 1183–1202 (2008)
Zaharias, P.: Usability in the Context of e-Learning: A Framework Augmenting ‘Traditional’ Usability Constructs with Instructional Design and Motivation to Learn. International Journal of Technology and Human Interaction 5(4), 38–61 (2009)
Brusilovsky, P., Eklund, J.: A study of user-model based link annotation in educational hypermedia. Journal of Universal Computer Science 4(4), 429–448 (1998)
Weibelzahl, S.: Evaluation of adaptive systems. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) UM 2001. LNCS (LNAI), vol. 2109, pp. 292–294. Springer, Heidelberg (2001)
Paramythis, A., Totter, A., Stephanidis, C.: A modular approach to the evaluation of adaptive user interfaces. In: Weibelzahl, S., Chin, D.N., Weber, G. (eds.) Empirical Evaluation of Adaptive Systems. In: Proceedings of Workshop at the Eighth, International Conference on User Modeling, UM 2001, Freiburg, p. 924 (2001)
Brusilovsky, P., Karagiannidis, C., Sampson, D.: Layered Evaluation of AdaptiveLearning Systems. International Journal of Continuing Engineering Education and Lifelong Learning, Special issue on Adaptivity in Web and Mobile Learning Services 14(4/5), 402–421 (2004)
Manouselis, N., Drachsler, H., Verbert, K., Duval, E.: Recommender Systems for Learning, pp. 1–61. Springer, Heidelberg (2012)
Verbert, K., Manouselis, N., Drachsler, H., Duval, E.: Dataset-driven Research to Support Learning and Knowledge Analytics. International Journal of Educational Technology & Society (2012)
Verbert, K., Drachsler, H., Manouselis, N., Wolpers, M., Vuorikari, R., Duval, E.: Dataset-driven Research for Improving Recommender Systems for Learning. In: 1st International Conference Learning Analytics & Knowledge, Banff, Alberta, Canada, February 27-March 1 (2011)
Rosas, S.R., Kane, M.: Quality and rigor of the concept mapping methodology: a pooled study analysis. Evaluation and Program Planning 35, 36–245 (2012)
Kane, M., Trochim, W.: Concept mapping for planning and evaluation. Sage Publishing, Thousand Oaks (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Drachsler, H., Stoyanov, S., d’Aquin, M., Herder, E., Guy, M., Dietze, S. (2014). An Evaluation Framework for Data Competitions in TEL. In: Rensing, C., de Freitas, S., Ley, T., Muñoz-Merino, P.J. (eds) Open Learning and Teaching in Educational Communities. EC-TEL 2014. Lecture Notes in Computer Science, vol 8719. Springer, Cham. https://doi.org/10.1007/978-3-319-11200-8_6
Download citation
DOI: https://doi.org/10.1007/978-3-319-11200-8_6
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-11199-5
Online ISBN: 978-3-319-11200-8
eBook Packages: Computer ScienceComputer Science (R0)