Advertisement

Electronic Portfolios Enhanced with Learning Analytics at the Workplace

  • M. F. van der SchaafEmail author
Living reference work entry

Abstract

During workplace-based learning, e.g., clinical or during an internship, supervisors’ quality of feedback and assessment is crucial for trainees’ expertise development. Electronic portfolios (ePortfolios) are often used as tools for longitudinal assessment at the workplace. So far, ePortfolios have not realized their full potential, since they are often not well tailored to the workplace and trainees’ needs. Potential data about trainees’ behavior at the workplace is generally underused, and the management of the data is complex. It is expected that ePortfolios enhanced with learning analytics may serve as means to improve the quality of feedback and assessment regarding trainees’ progress and development. This chapter addresses this by outlining an approach that is applied in a European 7th framework project (www.project-WatchMe.eu).

This chapter shows the development of an ePortfolio environment enhanced with learning analytics, to be used at the workplace in medical, veterinary, and teacher education. Evaluation took place by means of a quasi-experimental design regarding the impact of this environment on trainees’ motivation, their assessment experience, and their use. Data gathered in four institutes for medical, veterinary, and teacher education (n = 217) showed that trainees were highly motivated for their internships and positively evaluated the perceived feedback. The use of learning analytics features varied. In general visual feedback by means of a timeline of trainees’ progress was mostly used, while trainees barely used the features with written feedback. It is concluded that the promise of learning analytics connected to ePortfolios can only be fulfilled when developed and implemented through the eyes of the users.

Keywords

ePortfolio Learning analytics Workplace Feedback 

Notes

Acknowledgment

This study was conducted within the framework of the “Workplace-Based e-Assessment Technology for competency-Based Higher Multi-Professional Education” (WATCHME) project, supported by the European Commission 7th Framework Programme (grant agreement No. 619349). We thank all participants and members of the WATCHME project for their contributions: Charité Universitätsmedizin Berlin, Germany; Jayway, Denmark; Maastricht University, the Netherlands; Mateum BV., the Netherlands; NetRom Software SRL, Romania; Tartu Ülikool, Estonia; University of California San Francisco, USA; University Medical Center Utrecht, the Netherlands; University of Reading, UK; University of Veterinary Medicine, Hungary; and Utrecht University, the Netherlands.

References

  1. Agudo-Peregrina A, Iglesias-Pradas S, Conde-González MA, Hernández-García A (2014) Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning. Comput Hum Behav 31:542–550.  https://doi.org/10.1016/j.chb.2013.05.031CrossRefGoogle Scholar
  2. Barbera E (2009) Mutual feedback in e-portfolio assessment: an approach to the netfolio system. Br J Educ Technol 40(2):342–357.  https://doi.org/10.1111/j.1467-8535.2007.00803.xCrossRefGoogle Scholar
  3. Barrett HC (1998) Strategic questions: what to consider when planning for electronic portfolios. Learn Lead Technol 26:6–13Google Scholar
  4. Billett S (2004) Workplace participatory practices: conceptualising workplaces as learning environments. J Work Learn 16(6):312–324.  https://doi.org/10.1108/13665620410550295CrossRefGoogle Scholar
  5. Bryant LH, Chittum JR (2013) ePortfolio effectiveness: a(n ill-fated) search for empirical support. Int J ePortfolio 3(2):189–198Google Scholar
  6. Butler P (2006) A review of the literature on portfolios and electronic portfolios. Massey University College of Education, Palmerston NorthGoogle Scholar
  7. Dekker-Groen A, Van der Schaaf M, Stokking K (2012) Performance standards for teachers supporting nursing students’ reflection skills development. J Nurs Educ Pract 2(1):9–19.  https://doi.org/10.5430/jnep.vn1p9CrossRefGoogle Scholar
  8. Doyle GJ, Garrett B, Currie LM (2014) Integrating mobile devices into nursing curricula: opportunities for implementation using Rogers’ diffusion of innovation model. Nurse Educ Today 34(5):775–782.  https://doi.org/10.1016/j.nedt.2013.10.021CrossRefGoogle Scholar
  9. Duijn CC, Welink LS, Mandoki M, ten Cate OT, Kremer WD, Bok HG (2017) Am I ready for it? Students’ perceptions of meaningful feedback on entrustable professional activities. Perspect Med Educ 6:256.  https://doi.org/10.1007/s40037-017-0361-1CrossRefGoogle Scholar
  10. Dysthe O, Engelsen KS (2011) Portfolio practices in higher education in Norway in an international perspective: macro-, meso-and micro-level influences. Assess Eval High Educ 36(1):63–79.  https://doi.org/10.1080/02602930903197891CrossRefGoogle Scholar
  11. Elias T (2011) Learning analytics: definitions, processes and potential. Creative Commons Attribution 3.0Google Scholar
  12. Ericsson A, Pool R (2016) Peak: secrets from the new science of expertise. Harcourt, Houghton MifflinGoogle Scholar
  13. Ericsson KA, Charness N, Feltovich PJ, Hoffman RR (eds) (2006) The Cambridge handbook of expertise and expert performance. Cambridge University Press, New YorkGoogle Scholar
  14. Eynon B, Gambino LM, Török J (2014) What difference can ePortfolio make? A field report from the connect to learning project. Int J ePortfolio 4(1):95–114Google Scholar
  15. Feltovich PJ, Prietula MJ, Ericsson KA (2006) Studies of expertise from psychological perspectives. In: Ericsson KA, Charness N, Feltovich P, Hoffman RR (eds) Cambridge handbook of expertise and expert performance. Cambridge University Press, Cambridge, pp 41–67CrossRefGoogle Scholar
  16. Frank JR (2005) The CanMEDS 2005 physician competency framework. Better standards, better physicians. Better care, Ottawa, The Royal College of Physicians and Surgeons of CanadaGoogle Scholar
  17. Gibbs G, Simpson C (2003) Measuring the response of students to assessment: the assessment experience questionnaire. In: 11th Improving Student Learning Symposium, pp 1–12Google Scholar
  18. Greller W, Drachsler H (2012) Translating learning into numbers: a generic framework for learning analytics. J Educ Technol Soc 15(3):42–57Google Scholar
  19. Grossman P, Hammerness K, McDonald M (2009) Redefining teaching, re-imagining teacher education. Teach Teach Theory Pract 15(2):273–289.  https://doi.org/10.1080/13540600902875340CrossRefGoogle Scholar
  20. Hamp-Lyons L, Condon W (2000) Assessing the portfolio: principles for practice, theory, and research. Hampton Press, CresskillGoogle Scholar
  21. Hattie J, Timperley H (2007) The power of feedback. Rev Educ Res 77(1):81–112.  https://doi.org/10.3102/003465430298487CrossRefGoogle Scholar
  22. Jafari A, Kaufman C (eds) (2006) Handbook of research on eportfolios. IDEA Group Publishing, HersheyGoogle Scholar
  23. Jarafsky D, Martin JH (2008) Speech and language processing: an introduction to natural language processing, computational linguistics and speech recognition, 2nd edn. Prentice Hall, Upper Saddle RiverGoogle Scholar
  24. Jonker G, Hoff RG, Ten Cate O (2015) A case for competency-based anaesthesiology training with entrustable professional activities: an agenda for development and research. Eur J Anaesthesiol (EJA) 32(2):71–76.  https://doi.org/10.1097/EJA.0000000000000109CrossRefGoogle Scholar
  25. Kane MT (1992) An argument-based approach to validity. Psychol Bull 112(3):527CrossRefGoogle Scholar
  26. Kennelly E, Osborn D, Reardon R, Shetty B (2016) Guidance for ePortfolio researchers: a case study with implications for the ePortfolio domain. Int J ePortfolio 6(2):117–125. http://www.theijep.comGoogle Scholar
  27. Kovanović V, Joksimović S, Gašević D, Hatala M, Siemens G (2015) Content analytics: the definition, scope, and an overview of published research. In: Handbook of learning analytics. SoLAR, Edmonton.  https://doi.org/10.18608/hla17.007CrossRefGoogle Scholar
  28. Laskey KB (2008) MEBN: a language for first-order Bayesian knowledge bases. Artif Intell 172(2–3):140–178.  https://doi.org/10.1016/j.artint.2007.09.006CrossRefGoogle Scholar
  29. Leijen A, Slof B, Malva L, Hunt P, van Tartwijk JWF, van der Schaaf MF (2017) Performance-based competency requirements for learner teachers and how to assess them. Int J Inf Educ Technol 7(3):190–194.  https://doi.org/10.18178/ijiet.2017.7.3.864CrossRefGoogle Scholar
  30. McNamara DS, Allen LK, Crossley SA, Dascula M, Perret CA (2017) Natural language processing and learning analytics. In: Lang C, Siemens G, Wise A, Gašević D (eds) Handbook of learning analytics. SoLAR, Edmonton, pp 93–104.  https://doi.org/10.18608/hla17.008CrossRefGoogle Scholar
  31. Mislevy RJ, Behrens JT, Dicerbo KE, Levy R (2012) Design and discovery in educational assessment: evidence-centered design, psychometrics, and educational data mining. JEDM-J Educ Data Min 4(1):11–48Google Scholar
  32. Mulder M (2014) Conceptions of professional competence. In: Billett S, Harteis C, Gruber H (eds) International handbook of research in professional and practice-based learning. Springer, Dordrecht, pp 107–137Google Scholar
  33. Peacock S, Murray S, Scott A, Kelly J (2011) The transformative role of ePortfolios: feedback in healthcare learning. Int J ePortfolio 1(1):33–48. http://www.theijep.comGoogle Scholar
  34. Rezgui K, Mhiri H, Ghédira K (2014) Ontology-based e-Portfolio modeling for supporting lifelong competency assessment and development. Procedia Comput Sci 112:397–406.  https://doi.org/10.1016/j.proc.2017.08l041CrossRefGoogle Scholar
  35. Royal College of Veterinary Surgeons (RCVS) (2014) Practice standards scheme manual. Royal College of Veterinary Surgeons, LondonGoogle Scholar
  36. Ryan RM, Deci EL (2000) Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol 55(1):68.  https://doi.org/10.1037/110003-066X.55.1.68CrossRefGoogle Scholar
  37. Sadler DR (2010) Beyond feedback: developing student capability in complex appraisal. Assess Eval High Educ 35(5):535–550.  https://doi.org/10.1080/02602930903541015CrossRefGoogle Scholar
  38. Selwyn N (2015) Data entry: towards the critical study of digital data and education. Learn Media Technol 40(1):64–82.  https://doi.org/10.1080/17439884.2014.921628CrossRefGoogle Scholar
  39. Siemens G, Gašević D (2012) Guest editorial – learning and knowledge analytics. Educ Technol Soc 15(3):1–2Google Scholar
  40. Ten Cate O (2005) Entrustability of professional activities and competency-bases training. Med Educ 39:1176–1177CrossRefGoogle Scholar
  41. Ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, Van der Schaaf M (2015) Curriculum development for the workplace using entrustable professional activities (EPAs): AMEE guide no 99. Med Teach 37(11):983–1002CrossRefGoogle Scholar
  42. Toulmin SE (2003) The uses of argument. Updated edition. Cambridge University Press, Cambridge, UKGoogle Scholar
  43. Van der Schaaf MF, Stokking K, Verloop N (2008) Developing and validating a design for teacher portfolio assessment. Assess Eval High Educ 33(3):245–262.  https://doi.org/10.1080/02602930701292522CrossRefGoogle Scholar
  44. Van der Schaaf M, Baartman L, Prins F (2012) Exploring the role of assessment criteria during teachers’ collaborative judgement processes of students’ portfolios. Assess Eval High Educ 37(7):847–860CrossRefGoogle Scholar
  45. van der Schaaf M, Donkers J, Slof B, Moonen-van Loon J, van Tartwijk J, Driessen E, ... Ten Cate O (2017) Improving workplace-based assessment and feedback by an E-portfolio enhanced with learning analytics. Educ Technol Res Dev 65(2):359–380.  https://doi.org/10.1007/s11423-016-9496-8
  46. Van Schaik S, Plant J, O’sullivan P (2013) Promoting self-directed learning through portfolios in undergraduate medical education: the mentors’ perspective. Med Teach 35(2):139–144CrossRefGoogle Scholar
  47. Warren SJ, Lee J, Najmi A (2014) The impact of technology and theory on instructional design since 2000. In: Spector JM, Merrill MD, Elen\ J, Bishop MJ (eds) Handbook of research on educational communications and technology, 4th edn. Springer, New York, pp 89–99CrossRefGoogle Scholar
  48. Wisman-Zwarter N, Van der Schaaf M, Ten Cate OT, Jonker G, Van Klei WA, Hoff R (2016) Defining the content of Anaesthesiology training with entrustable professional activities. A delphi study. Eur J Anesthesiol 33(8):559–567CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University Medical Centre Utrecht and Utrecht UniversityUtrechtThe Netherlands
  2. 2.Center for Education and Department of EducationUtrecht UniversityUtrechtThe Netherlands

Section editors and affiliations

  • Esther Winther
    • 1
  1. 1.Vocational Education and TrainingUniversity of Duisburg-EssenDuisburgGermany

Personalised recommendations