Advertisement

Journal of Computing in Higher Education

, Volume 30, Issue 3, pp 572–598 | Cite as

The design, development, and implementation of student-facing learning analytics dashboards

  • Robert Bodily
  • Tarah K. Ikahihifo
  • Benjamin Mackley
  • Charles R. Graham
Article

Abstract

We have designed, developed, and implemented a student-facing learning analytics dashboard in order to support students as they learn in online environments. There are two separate dashboards in our system: a content recommender dashboard and a skills recommender dashboard. The content recommender helps students identify gaps in their content knowledge; the skills recommender helps students improve their metacognitive strategies. We discuss the technical requirements needed to develop a real-time student dashboard as well as report our inquiry into the functionality students want in a dashboard. The dashboards were evaluated with focus groups and a perceptions survey. Students were positive in their perceptions of the dashboards and 79% of the students that used the dashboards found them user-friendly, engaging, useful, and informative. One challenge encountered was low student use of the dashboard. Only 25% of students used the dashboard multiple times, despite favorable student perceptions of the dashboard. Additional research should examine how to motivate and support students to engage with dashboard feedback in online environments.

Keywords

Learning analytics Data visualization Student reporting tools Learning dashboards Iterative design Dashboard 

Notes

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

References

  1. Allen, I. E., & Seaman, J. (2014). Tracking online education in the United States, 1–45. Retrieved from http://www.onlinelearningsurvey.com/reports/gradechange.pdf. Accessed 18 Nov 2016.
  2. Anaya, A. R., Luque, M., & Peinado, M. (2016). A visual recommender tool in a collaborative learning experience. Expert Systems with Applications, 45, 248–259.CrossRefGoogle Scholar
  3. Arnold, K. E. (2010). Signals: Applying academic analytics. Educause Quarterly, 33(1), n1.Google Scholar
  4. Arnold, K. E., & Pistilli, M. D. (2012, April). Course signals at Purdue: Using learning analytics to increase student success. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 267–270). ACM.Google Scholar
  5. Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004, April). Off-task behavior in the cognitive tutor classroom: When students game the system. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 383–390). ACM.Google Scholar
  6. Bangert-Drowns, R. L., Kulik, C. L. C., Kulik, J. A., & Morgan, M. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61(2), 213–238.CrossRefGoogle Scholar
  7. Bekele, T. A. (2010). Motivation and satisfaction in internet-supported learning environments: A review. Educational Technology and Society, 13(2), 116–127.Google Scholar
  8. Bodily, R., & Verbert, K. (2017, March). Trends and issues in student-facing learning analytics reporting systems research. In Proceedings of the seventh international learning analytics & knowledge conference (pp. 309–318). ACM.Google Scholar
  9. Charleer, S., Klerkx, J., Odriozola, S., Luis, J., & Duval, E. (2013, December). Improving awareness and reflection through collaborative, interactive visualizations of badges. In ARTEL13: Proceedings of the 3rd workshop on awareness and reflection in technology-enhanced learning (vol. 1103, pp. 69–81). CEUR-WS.Google Scholar
  10. Corrin, L., & de Barba, P. (2015, March). How do students interpret feedback delivered via dashboards? In Proceedings of the fifth international conference on learning analytics and knowledge (pp. 430–431). ACM.Google Scholar
  11. Danado, J., Davies, M., Ricca, P., & Fensel, A. (2010, September). An authoring tool for user generated mobile services. In Future internet symposium (pp. 118–127). Berlin: Springer.CrossRefGoogle Scholar
  12. Doorn, D. J., Janssen, S., & O’Brien, M. (2010). Student attitudes and approaches to online homework. International Journal for the Scholarship of Teaching and Learning, 4(1), 5.CrossRefGoogle Scholar
  13. Elias, T. (2011). Learning analytics. Learning Analytics: Definitions, Processes and Potential. Retrieved June 2015, from http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf.
  14. Feild, J. (2015, June). Improving student performance using nudge analytics. In Proceedings of the 8th international conference on educational data mining (pp. 464–467).Google Scholar
  15. Ferguson, R., & Shum, S. B. (2012, April). Social learning analytics: Five approaches. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 23–33). ACM.Google Scholar
  16. Few, S. (2006). Information dashboard design: The effective visual communication of data. Sebastopol, CA: O’Reilly Media.Google Scholar
  17. Few, S. (2013). Information dashboard design: Displaying data for at-a-glance monitoring. Burlingame, CA: Analytics Press.Google Scholar
  18. Garrison, D. R. (2003). Cognitive presence for effective asynchronous online learning: The role of reflective inquiry, self-direction and metacognition. Elements of Quality Online Education: Practice and Direction, 4(1), 47–58.Google Scholar
  19. Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012, May). The student activity meter for awareness and self-reflection. In CHI’12 extended abstracts on human factors in computing systems (pp. 869–884). ACM.Google Scholar
  20. Govaerts, S., Verbert, K., Klerkx, J., & Duval, E. (2010, December). Visualizing activities for self-reflection and awareness. In International conference on web-based learning (pp. 91–100). Berlin: Springer.CrossRefGoogle Scholar
  21. Grann, J., & Bushway, D. (2014, March). Competency map: Visualizing student learning to promote student success. In Proceedings of the fourth international conference on learning analytics and knowledge (pp. 168–172). ACM.Google Scholar
  22. Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Educational Technology and Society, 15(3), 42–57.Google Scholar
  23. Hacker, D. J., Dunlosky, J., & Graesser, A. C. (Eds.). (1998). Metacognition in educational theory and practice. New York: Routledge.Google Scholar
  24. Harrer, A. (2015, July). A design proposal for learner-centered visualisations of learning analytics in collaborative scenarios. In 2015 IEEE 15th international conference on advanced learning technologies (ICALT) (pp. 208–210). IEEE.Google Scholar
  25. Harrer, A., & Göhnert, T. (2015, March). Integrated representations and small data: Towards contextualized and embedded analytics tools for learners. In Proceedings of the fifth international conference on learning analytics and knowledge (pp. 406–407). ACM.Google Scholar
  26. Hatziapostolou, T., & Paraskakis, I. (2010). Enhancing the impact of formative feedback on student learning through an online feedback system. Electronic Journal of e-Learning, 8(2), 111–122.Google Scholar
  27. Holman, C., Aguilar, S., & Fishman, B. (2013, April). GradeCraft: What can we learn from a game-inspired learning management system? In Proceedings of the third international conference on learning analytics and knowledge (pp. 260–264). ACM.Google Scholar
  28. Huizing, A., & Cavanagh, M. (2011). Planting contemporary practice theory in the garden of information science. Information Research Journal, 16(4), 497.Google Scholar
  29. Iandoli, L., Quinto, I., De Liddo, A., & Shum, S. B. (2014). Socially augmented argumentation tools: Rationale, design and evaluation of a debate dashboard. International Journal of Human-Computer Studies, 72(3), 298–319.CrossRefGoogle Scholar
  30. Jeon, J. H., Yeon, J., Lee, S. G., & Seo, J. (2014, January). Exploratory visualization of smartphone-based life-logging data using Smart Reality Testbed. In 2014 International conference on big data and smart computing (BIGCOMP) (pp. 29–33). IEEE.Google Scholar
  31. Jones, A., & Issroff, K. (2007). Learning technologies: Affective and social issues. In G. Conole & M. Oliver (Eds.), Contemporary perspectives in e-learning research: Themes, methods and impact on practice (pp. 190–202). London: Routledge.Google Scholar
  32. Jugo, I., Kovačić, B., & Slavuj, V. (2014, May). Using data mining for learning path recommendation and visualization in an intelligent tutoring system. In 2014 37th international convention on information and communication technology, electronics and microelectronics (MIPRO) (pp. 924–928). IEEE.Google Scholar
  33. Kerly, A., Ellis, R., & Bull, S. (2008). CALMsystem: A conversational agent for learner modelling. Knowledge-Based Systems, 21(3), 238–246.  https://doi.org/10.1016/j.knosys.2007.11.015.CrossRefGoogle Scholar
  34. Kim, J., Jo, I. H., & Park, Y. (2016). Effects of learning analytics dashboard: Analyzing the relations among dashboard utilization, satisfaction, and learning achievement. Asia Pacific Education Review, 17(1), 13–24.CrossRefGoogle Scholar
  35. Kortemeyer, G. (2015). An empirical study of the effect of granting multiple tries for online homework. American Journal of Physics, 83(7), 646–653.CrossRefGoogle Scholar
  36. Kuosa, K., Distante, D., Tervakari, A., Cerulo, L., Fernández, A., Koro, J., et al. (2016). Interactive visualization tools to improve learning and teaching in online learning environments. International Journal of Distance Education Technologies (IJDET), 14(1), 1–21.CrossRefGoogle Scholar
  37. Laffey, J. M., Amelung, C., & Goggins, S. (2014). Using analytics for activity awareness in learning systems. International Journal of Designs for Learning, 5(2), 101.CrossRefGoogle Scholar
  38. Lent, R. W., Brown, S. D., & Larkin, K. C. (1984). Relation of self-efficacy expectations to academic achievement and persistence. Journal of Counseling Psychology, 31(3), 356.CrossRefGoogle Scholar
  39. Manso-Vázquez, M., & Llamas-Nistal, M. (2015). Proposal of a learning organization tool with support for metacognition. IEEE Revista Iberoamericana de Tecnologias del Aprendizaje, 10(2), 35–42.CrossRefGoogle Scholar
  40. McAuley, J., O’Connor, A., & Lewis, D. (2012, April). Exploring reflection in online communities. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 102–110). ACM.Google Scholar
  41. Melero, J., Hernández-Leo, D., Sun, J., Santos, P., & Blat, J. (2015). How was the activity? A visualization support for a case of location-based learning design. British Journal of Educational Technology, 46(2), 317–329.CrossRefGoogle Scholar
  42. Millman, J. (1989). If at first you don’t succeed setting passing scores when more than one attempt is permitted. Educational Researcher, 18(6), 5–9.Google Scholar
  43. Muldner, K., Wixon, M., Rai, D., Burleson, W., Woolf, B., & Arroyo, I. (2015). Exploring the impact of a learning dashboard on student affect. In  C. Conati, N. Heffernan, A. Mitrovic, M. F. Verdejo (Eds.), International conference on artificial intelligence in education (pp. 307–317). Berlin: Springer.CrossRefGoogle Scholar
  44. Odriozola, S., Luis, J., Verbert, K., Govaerts, S., & Duval, E. (2011, August). Visualizing PLE usage. In Proceedings of EFEPLE11 1st workshop on exploring the fitness and evolvability of personal learning environments (Vol. 773, pp. 34–38). CEUR WS.Google Scholar
  45. Olmos, M. M., & Corrin, L. (2012). Learning analytics: A case study of the process of design of visualizations. Journal of Asynchronous Learning Networks, 16(3), 39–49.Google Scholar
  46. Ott, C., Robins, A., Haden, P., & Shephard, K. (2015). Illustrating performance indicators and course characteristics to support students’ self-regulated learning in CS1. Computer Science Education, 25(2), 174–198.CrossRefGoogle Scholar
  47. Park, Y., & Jo, I. H. (2015). Development of the learning analytics dashboard to support students’ learning performance. Journal of Universal Computer Science, 21(1), 110–133.Google Scholar
  48. Resnick, P., & Varian, H. R. (1997). Recommender systems. Communications of the ACM, 40(3), 56–58.CrossRefGoogle Scholar
  49. Ruipérez-Valiente, J. A., Muñoz-Merino, P. J., & Kloos, C. D. (2013, November). An architecture for extending the learning analytics support in the Khan Academy framework. In Proceedings of the first international conference on technological ecosystem for enhancing multiculturality (pp. 277–284). ACM.Google Scholar
  50. Ruipérez-Valiente, J. A., Muñoz-Merino, P. J., Leony, D., & Kloos, C. D. (2015). ALAS-KA: A learning analytics extension for better understanding the learning process in the Khan Academy platform. Computers in Human Behavior, 47, 139–148.CrossRefGoogle Scholar
  51. Rydberg, E. (2011). Visualizing the software development process by analyzing software engineering data. Stockholm: Royal Institute of Technology.Google Scholar
  52. Santos, J. L., Govaerts, S., Verbert, K., & Duval, E. (2012, April). Goal-oriented visualizations of activity tracking: A case study with engineering students. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 143–152). ACM.Google Scholar
  53. Santos, J. L., Verbert, K., & Duval, E. (2012b). Empowering students to reflect on their activity with StepUp!: Two case studies with engineering students (p. 73). Editorial: Awareness and Reflection in Technology Enhanced Learning.Google Scholar
  54. Santos, J. L., Verbert, K., Govaerts, S., & Duval, E. (2013, April). Addressing learner issues with StepUp!: An evaluation. In Proceedings of the third international conference on learning analytics and knowledge (pp. 14–22). ACM.Google Scholar
  55. Santos, J., Verbert, K., Klerkx, J., Duval, E., Charleer, S., & Ternier, S. (2015). Tracking data in open learning environments. Journal of Universal Computer Science, 21(7), 976–996.Google Scholar
  56. Schmitz, H. C., Scheffel, M., Friedrich, M., Jahn, M., Niemann, K., & Wolpers, M. (2009, September). CAMera for PLE. In European conference on technology enhanced learning (pp. 507–520). Berlin: Springer.Google Scholar
  57. Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., et al. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30–41.CrossRefGoogle Scholar
  58. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.CrossRefGoogle Scholar
  59. Siemens, G. (2010, July 22). 1st International conference on learning analytics and knowledge 2011. Retrieved March 30, 2016, from https://tekri.athabascau.ca/analytics/.
  60. Silius, K., Miilumäki, T., Huhtamäki, J., Tebest, T., Meriläinen, J., & Pohjolainen, S. (2010). Students’ motivations for social media enhanced studying and learning. Knowledge Management and E-Learning: An International Journal, 2(1), 51–67.Google Scholar
  61. Silius, K., Tervakari, A. M., & Kailanto, M. (2013, March). Visualizations of user data in a social media enhanced web-based environment in higher education. In 2013 IEEE global engineering education conference (EDUCON) (pp. 893–899). IEEE.Google Scholar
  62. Stallings, J. (1980). Allocated academic learning time revisited, or beyond time on task. Educational researcher, 9(11), 11–16.CrossRefGoogle Scholar
  63. Steel, P. (2007). The nature of procrastination: A meta-analytic and theoretical review of quintessential self-regulatory failure. Psychological Bulletin, 133(1), 65–94.  https://doi.org/10.1037/0033-2909.133.1.65.CrossRefGoogle Scholar
  64. Tervakari, A. M., Silius, K., Koro, J., Paukkeri, J., & Pirttila, O. (2014, April). Usefulness of information visualizations based on educational data. In 2014 IEEE global engineering education conference (EDUCON) (pp. 142–151). IEEE.Google Scholar
  65. Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dashboard applications. American Behavioral Scientist, 57(10), 1–10.  https://doi.org/10.1177/0002764213479363.CrossRefGoogle Scholar
  66. Verbert, K., Govaerts, S., Duval, E., Santos, J. L., Van Assche, F., Parra, G., et al. (2014). Learning dashboards: An overview and future research opportunities. Personal and Ubiquitous Computing, 18(6), 1499–1514.  https://doi.org/10.1007/s00779-013-0751-2.CrossRefGoogle Scholar
  67. Wilson, B. G. (2013). A practice-centered approach to instructional design. In Learning, problem solving, and mind tools: Essays in honor of David H. Jonassen, (pp. 35–54).Google Scholar
  68. Yoo, Y., Lee, H., Jo, I., & Park, Y. (2015). Educational dashboards for smart learning: Review of case studies. Emerging Issues in Smart Learning.  https://doi.org/10.1007/978-3-662-44188-6.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Brigham Young UniversityProvoUSA

Personalised recommendations