Abstract
The study contributes to training evaluation research by systematically comparing training outcomes between corporate e-learning and classroom training. We investigated a sample of vocational trainees (N = 86) in a field experiment with a time-lag design and examined their subjective as well as objective training success. While training success was subjectively perceived to vary in effectiveness depending on the setting right after the training, with higher scores for trainees in classroom training, it was perceived similarly effective in both training settings after six to eight weeks. With regard to objective training success, we found that whereas declarative and procedural knowledge scores increased for e-learning trainees, they decreased for trainees in classroom training. However, strengths of e-learning become more prevalent six to eight weeks after the training. Thus in the end, e-learning is as effective as classroom training for procedural knowledge and more effective than classroom training for declarative knowledge as e-learning trainees catch up on previous differences across time. We conclude that it is not the training setting or delivery media per se that leads to higher levels of trainings success in the long run but in order to be effective, trainings should be designed taking especially the type of learning content (declarative or procedural knowledge) into account.
Similar content being viewed by others
Notes
We consequently use the term (corporate) e-learning when referring to virtual training, web-based training, or e-learning.
A large training setting effect for satisfaction (p < .001, η2partial = .15); moderate training setting effects for utility (p = .02, η2partial = .07), knowledge (p = .02, η2partial = .06), self-efficacy (p = .002, η2partial = .11), application to practice (p = .02, η2partial = .07), organizational results (p < .001, η2partial = .12).
There was a substantial large main effect on the combined objective trainings success scales for training setting, Wilks’ Lambda = .77, F(2, 83) = 12.48, p < .001, η2partial = .23. The main effect for time was not significant, Wilks’ Lambda = .97, F(2, 83) = 1.48, p = .23, η2partial = .03.
For declarative knowledge, a large interaction effect was found, p < .001, η2partial = .17, with no main effect for training setting, p = .20, η2partial = .02, and no main effect for time, p = .65, η2partial = .002. For procedural knowledge, a large interaction effect was found, p < .001, η2partial = .28, with a moderate main effect for training setting, p < .001, η2partial = .13, and no main effect for time, p = .25, η2partial = .02.
References
Aguinis, H., & Kraiger, K. (2009). Benefits of training and development for individuals and teams, organizations, and society. Annual Review of Psychology, 60, 451–474. https://doi.org/10.1146/annurev.psych.60.110707.163505.
Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick’s levels of training criteria: thirty years later. Personnel Psychology, 42, 331–342. https://doi.org/10.1111/j.1744-6570.1989.tb00661.x.
Alliger, G. M., Tannenbaum, S. I., Bennett, W., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341–358. https://doi.org/10.1111/j.1744-6570.1997.tb00911.x.
American Society for Training and Development (ASTD). (2000). E-learning evaluation method gains support in Canada. ASTD Learning Circuits. Retrieved from www.learningcircuits.org.
American Society for Training and Development (ASTD). (2012). The 2012 state of the industry report. Retrieved from http://store.astd.org/Default.aspx?tabid=%20167&ProductId=23805.
American Society for Training and Development (ASTD). (2014). The 2014 state of the industry report. Retrieved from http://files.astd.org/Research/LookInsides/2014SOIR LookInside.pdf.
Arthur Jr., W., Bennett Jr., W., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 88, 234–245. https://doi.org/10.1037/0021-9010.88.2.234.
Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology, 41, 63–105. https://doi.org/10.1111/j.1744-6570.1988.tb00632.x.
Bandura, A. (Ed.). (1995). Self-efficacy in changing societies. Cambridge: Cambridge University Press.
Bates, R. A., & Khasawneh, S. (2005). Organizational learning culture, learning transfer climate and perceived innovation in Jordanian organizations. International Journal of Training and Development, 9, 96–109. https://doi.org/10.1111/j.1468-2419.2005.00224.x.
Bell, B. S., & Federman, J. E. (2013). E-learning in postsecondary education. The Future of Children, 23, 165–185. https://doi.org/10.1353/foc.2013.0007.
Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., ... Huang, B. (2004). How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74, 379–439. https://doi.org/10.3102/00346543074003379.
Blume, B. D., Ford, J. K., Baldwin, T. T., & Huang, J. L. (2010). Transfer of training: a meta-analytic review. Journal of Management, 36, 1065–1105. https://doi.org/10.1177/0149206309352880.
Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: a simple proposal with multiple implications. Review of Research in Education, 24, 61–100. https://doi.org/10.3102/0091732X024001061.
Bürg, O., & Mandl, H. (2005). Akzeptanz von E-learning in Unternehmen [Acceptance of e-learning in companies]. Zeitschrift für Personalpsychologie, 4, 75–85. https://doi.org/10.1026/1617-6391.4.2.75.
Burgess, J. R., & Russell, J. E. (2003). The effectiveness of distance learning initiatives in organizations. Journal of Vocational Behavior, 63, 289–303. https://doi.org/10.1016/S0001-8791(03)00045-9.
Burke, L. A., & Hutchins, H. M. (2007). Training transfer: An integrative literature review. Human Resource Development Review, 6, 263–296. https://doi.org/10.1177/1534484307303035.
Burke, L. A., Hutchins, H. M., & Saks, A. (2013). Best practices in training transfer. In M. Paludi (Ed.), The psychology for business success (Vol. 3, pp. 115–132). Westport: Praeger.
Cavanaugh, C. S. (2001). The effectiveness of interactive distance education technologies in K-12 learning: A meta-analysis. International Journal of Educational Telecommunications, 7, 73–88.
Chen, H. C., Holton, E. F., & Bates, R. A. (2005). Development and validation of the learning transfer system inventory in Taiwan. Human Resource Development Quarterly, 16, 55–84. https://doi.org/10.1002/hrdq.1124.
Cohen, J. (1988). Statistical power analysis for the behavioral science (2nd ed.). Hillsdale: Erlbaum.
Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of training motivation: a meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85, 678–707. https://doi.org/10.1037/0021-9010.85.5.678.
DeRouin, R. E., Fritzsche, B. A., & Salas, E. (2005). E-learning in organizations. Journal of Management, 31, 920–940. https://doi.org/10.1177/0149206305279815.
Docebo. (2014). E-learning market trends & forecast 2014–2016 report. Retrieved from https://www.docebo.com/landing/contactform/elearning-market-trends-and-forecast-2014-2016-docebo-report.pdf.
Ebbinghaus, H. (2013). Memory: A contribution to experimental psychology. Annals of Neurosciences, 20, 155–156. https://doi.org/10.5214/ans.0972.7531.200408.
Ferguson, R. (2012). Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4, 304–317. https://doi.org/10.1504/IJTEL.2012.051816.
Fry, K. (2001). E-learning markets and providers: Some issues and prospects. Education and Training, 43, 233–239. https://doi.org/10.1108/EUM0000000005484.
Gibson, C. B., & Gibbs, J. L. (2006). Unpacking the concept of virtuality: The effects of geographic dispersion, electronic dependence, dynamic structure, and national diversity on team innovation. Administrative Science Quarterly, 51, 451–495.
Gill, S. (2017). From a training culture to a learning culture. Retrieved from https://www.td.org/Publications/Blogs/Human-Capital-Blog/2017/01/From-a-Training-Culture-to-a-Learning-Culture.
Gnefkow, T. (2008). Lerntransfer in der betrieblichen Weiterbildung: Determinanten für den Erfolg externer betrieblicher Weiterbildungen im Lern- und Funktionsfeld aus Teilnehmerperspektive [Transfer of learning in vocational training: Determinants for the success of external vocational training in the learning and functional area from participants’ perspective] (doctoral dissertation). Retrieved from http://pub.uni-bielefeld.de/luur/download?func=downloadFile&recordOId=2303980&fileOId=2303983.
Godden, D. R., & Baddeley, A. D. (1975). Context-dependent memory in two natural environments: on land and underwater. British Journal of Psychology, 66, 325–331. https://doi.org/10.1111/j.2044-8295.1975.tb01468.x.
Goldstein, I. L., & Ford, J. K. (2002). Training in organizations: Needs assessment, development, and evaluation (4th ed.). Belmont: Wadsworth.
Gravill, J., & Compeau, D. (2008). Self-regulated learning strategies and software training. Information Management, 45, 288–296. https://doi.org/10.1016/j.im.2008.03.001.
Grohmann, A., & Kauffeld, S. (2013). Evaluating training programs: development and correlates of the questionnaire for professional training evaluation. International Journal of Training and Development, 17, 135–155. https://doi.org/10.1111/ijtd.12005.
Grossman, R., & Salas, E. (2011). The transfer of training: what really matters. International Journal of Training and Development, 15, 103–120. https://doi.org/10.1111/j.1468-2419.2011.00373.x.
Hamtini, T. M. (2008). Evaluating e-learning programs: an adaptation of Kirkpatrick's model to accommodate e-learning environments. Journal of Computer Science, 4, 693–698. https://doi.org/10.3844/jcssp.2008.693.698.
Han, H.-J., Dick, G., Case, T., & Van Slyke, C. (2009). Key capabilities, components, and evolutionary trends in corporate e-learning systems. In H. Song & T. T. Kidd (Eds.), Handbook of research on human performance and instructional technology (pp. 446–469). https://doi.org/10.4018/978-1-60566-782-9.ch027.
Hansford, B. C., & Hattie, J. A. (1982). The relationship between self and achievement/performance measures. Review of Educational Research, 52, 123–142. https://doi.org/10.3102/00346543052001123.
Hochholdinger, S., & Beinicke, A. (2012). Potenziale und Herausforderungen netzbasierten Lernens [Potentials and challenges of net-based learning]. Personal Quarterly, 64, 16–23.
Holton, E. F. (1996). The flawed four-level evaluation model. Human Resource Development Quarterly, 7, 5–21. https://doi.org/10.1002/hrdq.3920070103.
Holton, E. F. (2005). Holton’s evaluation model: new evidence and construct elaborations. Advances in Developing Human Resources, 7, 37–54. https://doi.org/10.1177/1523422304272080.
Holton, E. F., Bates, R. A., & Ruona, W. E. A. (2000). Development of a generalized learning transfer system inventory. Human Resource Development Quarterly, 11, 333–360. https://doi.org/10.1002/1532-1096(200024)11:4<333::AID-HRDQ2>3.0.CO;2-P.
Ibis Capital. (2013). Global e-learning investment review [Report]. Retrieved from http://edxusgroup.com/wp-content/uploads/2013/12/13-01-08-IBIS-Capital-e-Learning-Lessons-for-the-Future.pdf.
International Labour Organization. (2011). A skilled workforce for strong, sustainable and balanced growth: A G20 training strategy. Retrieved from http://www.oecd.org/g20/topics/employment-and-social-policy/G20-Skills-Strategy.pdf.
Kaltenbaek, J. (2003). E-Learning und Blended-Learning in der betrieblichen Weiterbildung: Möglichkeiten und Grenzen aus Sicht von Mitarbeitern und Personalverantwortlichen in Unternehmen [E-learning and blended-learning in corporate training: Chances and limits from the perspective of employees and T&D professionals] [data file]. Retrieved from http://www.weissensee-verlag.de/autoren/Kaltenbaek/Kaltenbaek-kurz.pdf.
Kauffeld, S. (2010). Nachhaltige Weiterbildung. Betriebliche Seminare und Trainings entwickeln, Erfolge messen, Transfer sichern [Sustainable training. Develop seminars and trainings, measure results, secure transfer]. https://doi.org/10.1007/978-3-540-95954-0.
Kauffeld, S., Bates, R. A., Holton, E. F., & Müller, A. C. (2008). Das deutsche Lerntransfer-System-Inventar (GLTSI): Psychometrische Überprüfung der deutschsprachigen Version [The German version of the Learning Transfer Systems Inventory (GLTSI): Psychometric validation]. Zeitschrift für Personalpsychologie, 7, 50–69. https://doi.org/10.1026/1617-6391.7.2.50.
Kauffeld, S., Brennecke, J., & Strack, M. (2009). Erfolge sichtbar machen: Das Maßnahmen-Erfolgs-Inventar (MEI) zur Bewertung von Trainings [Visualizing training outcomes: The MEI for training evaluations]. In S. Kauffeld, S. Grote, & E. Frieling (Eds.), Handbuch Kompetenzentwicklung (pp. 55–78). Stuttgart: Schäffer-Poeschel.
Kauffeld, S., Lorenzo, G., & Weisweiler, S. (2012). Wann wird Weiterbildung nachhaltig? Erfolg und Erfolgsfaktoren bei Lerntransfer [When do further trainings become sustainable? Success and factors of success in transfer of training]. Personal Quarterly, 64, 10–15.
Kimiloglu, H., Ozturan, M., & Kutlu, B. (2017). Perceptions about and attitude toward the usage of e-learning in corporate training. Computers in Human Behavior, 72, 339–349. https://doi.org/10.1016/j.chb.2017.02.062.
Kirkpatrick, D. L. (1959). Techniques for evaluating training programs. Journal of American Society of Training Directors, 13, 21–26.
Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels. San Francisco: Berrett-Koehler.
Kosarzycki, M. P., Salas, E., DeRouin, R., & Fiore, S. M. (2003). Distance learning in organizations: a review and assessment of future needs. Advances in Human Performance and Cognitive Engineering Research, 3, 69–98. https://doi.org/10.1016/S1479-3601(02)03004-7.
Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311–328. https://doi.org/10.1037/0021-9010.78.2.311.
Kumkale, G. T., & Albarracín, D. (2004). The sleeper effect in persuasion: a meta-analytic review. Psychological Bulletin, 130, 143–172. https://doi.org/10.1037/0033-2909.130.1.143.
Kurtz-Costes, B. E., & Schneider, W. (1994). Self-concept, attributional beliefs, and school achievement: a longitudinal analysis. Contemporary Educational Psychology, 19, 199–216. https://doi.org/10.1006/ceps.1994.1017.
Lin, J., & Cantoni, L. (2017). Assessing the performance of a tourism MOOC using the Kirkpatrick model: A supplier’s point of view. In R. Schegg & B. Stangl (Eds.), Information and communication technologies in tourism 2017 (pp. 129–142). https://doi.org/10.1007/978-3-319-51168-9.
Machtmes, K., & Asher, J. W. (2000). A meta-analysis of the effectiveness of telecourses in distance education. American Journal of Distance Education, 14, 27–46. https://doi.org/10.1080/08923640009527043.
Macpherson, A., Elliot, M., Harris, I., & Homan, G. (2004). E-learning: reflections and evaluation of corporate programmes. Human Resource Development International, 7, 295–313. https://doi.org/10.1080/13678860310001630638.
Means, B., Toyama, Y., Murphy, R. F., Baki, M., & Jones, K. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. (Research Report No. ED-04-CO-0040 Task 0006). Retrieved from U.S. Department of Education, Office of Planning, Evaluation, and Policy Development website: https://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf.
MMB-Institute for Media and Competence Research. (2010). Telefonische Befragung zum Einsatz von eLearning in deutschen Großunternehmen [Telephone survey on the implementation of e-learning in large German companies] [Report]. Retrieved from http://www.mmb-institut.de/projekte/digitales-lernen/Einsatz-von-E-Learning-in-deutschen-Grossunternehmen.pdf.
Newton, R., & Doonga, N. (2007). Corporate e-learning: Justification for implementation and evaluation of benefits. A study examining the views of training managers and training providers. Education for Information, 25, 111–130.
Phillips, J. J. (1997). Measuring return on investment. Alexandria: ASTD.
Podsakoff, P. M., MacKenzie, S. B., Lee, J., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903. https://doi.org/10.1037/0021-9010.88.5.879.
Rey, G. D. (2009). E-Learning. Theorien, Gestaltungsempfehlungen und Forschung [E-learning. Theories, recommendations for design, and research]. Bern: Huber.
Saks, A. M. (2002). So what is a good transfer of training estimate? A reply to Fitzpatrick. The Industrial-Organizational. Psychologist, 39, 29–30. https://doi.org/10.1037/e576922011-004.
Saks, A. M., Salas, E., & Lewis, P. (2014). The transfer of training. International Journal of Training and Development, 18, 81–83. https://doi.org/10.1111/ijtd.12032.
Salas, E., Tannenbaum, S. I., Kraiger, K., & Smith-Jentsch, K. A. (2012). The science of training and development in organizations: what matters in practice. Psychological Science in the Public Interest, 13, 74–101. https://doi.org/10.1177/1529100612436661.
Schacter, D. L. (2001). The seven sins of memory: How the mind forgets and remembers. New York: Houghton Mifflin Harcourt.
Schneider, W. (2008). Expertiseerwerb [Acquisition of expertise]. In W. Schneider & M. Hasselhorn (Eds.), Handbuch der Pädagogischen Psychologie (pp. 136–144). Göttingen: Hogrefe.
Schwartz, D. L., Bransford, J. D., & Sears, D. (2005). Efficiency and innovation in transfer. In J. P. Mestre (Ed.), Transfer of learning from a modern multidisciplinary perspective (pp. 1–51). Greenwich: IAP.
Schwartz, D. L., Chase, C. C., & Bransford, J. D. (2012). Resisting overzealous transfer: coordinating previously successful routines with needs for new learning. Educational Psychologist, 47, 204–214. https://doi.org/10.1080/00461520.2012.696317.
Shekhar, S. (2006). Understanding the virtuality of virtual organizations. Leadership and Organization Development Journal, 27, 465–483. https://doi.org/10.1108/01437730610687755.
Shen, N., Yufe, S., Saadatfard, O., Sockalingam, S., & Wiljer, D. (2017). Rebooting Kirkpatrick: Integrating information system theory into the evaluation of web-based continuing professional development interventions for interprofessional education. Journal of Continuing Education in the Health Professions, 37, 137–146. https://doi.org/10.1097/CEH.0000000000000154.
Sitzmann, T. (2011). A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Personnel Psychology, 64, 489–528. https://doi.org/10.1111/j.1744-6570.2011.01190.x.
Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. A. (2006). The comparative effectiveness of web-based and classroom instruction: a meta-analysis. Personnel Psychology, 59, 623–664. https://doi.org/10.1111/j.1744-6570.2006.00049.x.
Smith, S. M. (1994). Theoretical principles of context-dependent memory. In M. Gruneberg & P. E. Morris (Eds.), Theoretical aspects of memory (2nd ed., pp. 168–195). https://doi.org/10.4324/9780203978108.
Smith, S. M., & Vela, E. (2001). Environmental context-dependent memory: a review and meta-analysis. Psychonomic Bulletin & Review, 8, 203–220. https://doi.org/10.3758/BF03196157.
Solga, M. (2011). Förderung von Lerntransfer [Promoting learning transfer]. In J. Ryschka, M. Solga, & M. Mattenklott (Eds.), Praxishandbuch Personalentwicklung. Instrumente, Konzepte, Beispiele (pp. 339–368). https://doi.org/10.1007/978-3-8349-6384-0_6.
Spector, P. E. (2006). Method variance in organizational research: truth or urban legend? Organizational Research Methods, 9, 221–232. https://doi.org/10.1177/1094428105284955.
Strother, J. B. (2002). An assessment of the effectiveness of e-learning in corporate training programs. The International Review of Research in Open and Distributed Learning, 3(1).
Tharenou, P., Saks, A. M., & Moore, C. (2007). A review and critique of research on training and organizational-level outcomes. Human Resource Management Review, 17, 251–273. https://doi.org/10.1016/j.hrmr.2007.07.004.
Tracey, J. B., Tannenbaum, S. I., & Kavanagh, M. J. (1995). Applying trained skills on the job: the importance of the work environment. Journal of Applied Psychology, 80, 239–252. https://doi.org/10.1037/0021-9010.80.2.239.
Tracey, J. B., Murphy, J., & Horton-Tognazzini, L. (2016). A framework for evaluating MOOCs in applied hospitality and tourism settings. In A. Inversini & R. Schegg (Eds.), Information and Communication Technologies in Tourism 2016 (pp. 667–679). https://doi.org/10.1007/978-3-319-28231-2_48.
Twitchell, S., Holton, E. F., & Trott, J. W. (2000). Technical training evaluation practices in the United States. Performance Improvement Quarterly, 13, 84–109. https://doi.org/10.1111/j.1937-8327.2000.tb00177.x.
UC San Diego. (2015, September 15). What is continuous innovation. Retrieved from https://blink.ucsd.edu/sponsor/innovate/whatis.html.
United Nations Educational, Scientific and Cultural Organization (UNESCO). (2005). Towards knowledge societies [Report]. Retrieved from http://unesdoc.unesco.org/images/0014/001418/141843e.pdf.
Van Buren, M. E., & Erskine, W. (2002). The 2002 ASTD state of the industry report. Alexandria: American Society of Training and Development Press.
Wang, G. G., & Wilcox, D. (2006). Training evaluation: knowing more than is practiced. Advances in Developing Human Resources, 8, 528–539. https://doi.org/10.1177/152342230629.
Welsh, E. T., Wanberg, C. R., Brown, K. G., & Simmering, M. J. (2003). E-learning: emerging uses, empirical results and future directions. International Journal of Training and Development, 7, 245–258. https://doi.org/10.1046/j.1360-3736.2003.00184.x.
Zhao, Y., Lei, J., Yan, B., Lai, C., & Tan, S. (2005). What makes the difference? A practical analysis of research on the effectiveness of distance education. Teachers College Record, 107, 1836–1884. https://doi.org/10.1111/j.1467-9620.2005.00544.x.
Zhao, J., Qi, Z., & De Pablos, P. O. (2014). Enhancing enterprise training performance: Perspectives from knowledge transfer and integration. Computers in Human Behavior, 30, 567–573. https://doi.org/10.1016/j.chb.2013.06.041.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Beinicke, A., Bipp, T. Evaluating Training Outcomes in Corporate E-Learning and Classroom Training. Vocations and Learning 11, 501–528 (2018). https://doi.org/10.1007/s12186-018-9201-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12186-018-9201-7