A Benchmark Collection for Mapping Program Educational Objectives to ABET Student Outcomes: Accreditation

  • Addin Osman
  • Anwar Ali Yahya
  • Mohammed Basit Kamal
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 753)


This research aims to present a collection of dataset, which represents the mapping of program education objectives to the ABET student outcomes. The dataset has been collected by the authors from 32 self-study reports from Engineering programs accredited by ABET, which are available online. The paper presents the constraints under which, the dataset was produced, because its understanding plays a vital role in the usage of this collection in future researches. To illustrate the properties and usefulness of the collection, the dataset has been cleansed, preprocessed, some features have been selected, then it has been benchmarked using nine of the widely used supervised multiclass classification techniques (Binary Relevance, Label Powerset, Classifier Chains, Pruned Sets, Random k-label sets, Ensemble of Classifier Chains, Ensemble of Pruned Sets, Multi-Label k Nearest Neighbors and Back-Propagation Multi-Label Learning). The techniques have been compared to each other using five well-known measurements (Accuracy, Hamming Loss, Micro-F, Macro-F, and Macro-F). The Ensemble of Classifier Chains and Ensemble of Pruned Sets have achieved encouraging performance compared to the other experimented multi-label classification methods. The Classifier Chains method has shown the worst performance. In general, promising results have been achieved. New research directions and baseline experimental results for future studies in educational data mining in general and in accreditation in specific have been provided.


Benchmark collection Program educational objectives Student outcomes ABET Accreditation Machine learning Supervised multiclass classification Text mining 


  1. 1.
    Fabrizio, S.: Machine learning in automated text categorization. ACM Comput. Surv. 34(1), 1–47 (2002)CrossRefGoogle Scholar
  2. 2.
    Shweta, C.D., Maya, I., Parag, K.: Empirical studies on machine learning based text classification algorithms. Adv. Comput. Int. J. (ACIJ) 2(6), 161–169 (2011)CrossRefGoogle Scholar
  3. 3.
    Fabricio, A.B., Daniel, C., Guimarães P.: Combined unsupervised and semi-supervised learning for data classification. In: IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), Salerno, Italy, pp. 13–16 (2016)Google Scholar
  4. 4.
    Lunke, F., Yong, X., Xiaozhao, F., Jian, Y.: Low rank representation with adaptive distance penalty for semi-supervised subspace classification. Pattern Recogn. 67, 252–262 (2017). Scholar
  5. 5.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Secaucus (2006)zbMATHGoogle Scholar
  6. 6.
    Murphy, K.P.: Machine Learning: A Probabilistic Perspective, 1st edn. The MIT Press, Cambridge (2012)zbMATHGoogle Scholar
  7. 7.
    Duda, R.O., Hart, P., Stork, D.: Pattern Classification, 2nd edn. Wiley-Interscience, New York (2000)zbMATHGoogle Scholar
  8. 8.
    David, D.L., Robert, E.S., James, P.C., Ron, P.: Training algorithms for linear text classifiers. In: Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1996), pp. 298–306. ACM, New York (1996)Google Scholar
  9. 9.
    David, D.L.: Reuters-21578 text Categorization test collection. Distribution 1.0. Readme file (version 1.2). Manuscript (1997)Google Scholar
  10. 10.
    Yiming, Y.: An evaluation of statistical approaches to text categorization. Inf. Retrieval 1(1–2), 67–88 (1999)Google Scholar
  11. 11.
    David, D.L., Yiming, Y., Tony, G.R., Fan, L.: RCV1: a new benchmark collection for text categorization research. J. Mach. Learn. Res. 5, 361–397 (2004)Google Scholar
  12. 12.
    Pratiksha, Y., Gawande, S.H.: A comparative study on different types of approaches to text categorization. Int. J. Mach. Learn. Comput. 2(4), 423–426 (2012)Google Scholar
  13. 13.
    ABET, ABET Strategic Plan, Accreditation Board for Engineering and Technology, Inc., ABET, 1 November 1997Google Scholar
  14. 14.
    Engineering Accreditation Commission (ABET), Criteria for Accrediting Engineering Programs Effective for Review During the 2015–2016 Accreditation Cycle, 415 N. Charles Street Baltimore, MD 21201, United States of Ameriaca, ABET (2014)Google Scholar
  15. 15.
    ABET, Criteria for Accrediting Engineering Programs Effective for Reviews During the 2016–2017 Accrediting CycleGoogle Scholar
  16. 16.
    de Baker, R.S.J.: Data mining for education. In: McGaw, B., Peterson, P., Baker, E. (eds.) International Encyclopedia of Education, 3rd edn. Elsevier, Oxford (2010)Google Scholar
  17. 17.
    Romero, C., Ventura, S.: Educational data mining: a survey from 1995 to 2005. Expert Syst. Appl. 33(1), 135–146 (2007)CrossRefGoogle Scholar
  18. 18.
    de Baker, R.S.J., Yacef, K.: The state of educational data mining in 2009: a review and future vision. J. Educ. Data Min. 1(1), 1–15 (2009)Google Scholar
  19. 19.
    Peña-Ayala, A., Domínguez, R., Medel, J.: Educational data mining: a sample of review and study case. World J. Educ. Technol. 2, 118–139 (2009)Google Scholar
  20. 20.
    Romero, C., Ventura, S.: Educational data mining: a review of the state of the art. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 40(6), 601–618 (2010)CrossRefGoogle Scholar
  21. 21.
    Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S.H., Isohanni, E., Korhonen, A., Petersen, A., Rivers, K., Rubio, M.Á., Sheard, J., Skupas, B., Spacco, J., Szabo, C., Toll, D.: Educational data mining and learning analytics in programming: Literature review and case studies. In: Proceedings of the 2015 ITiCSE on Working Group Reports, Annual Conference on Innovation and Technology in Computer Science Education, pp. 41–63. ACM (2015).
  22. 22.
    Fatima, D., Fatima, S., Prasad, A.V.K.: A survey on research work in educational data mining. J. Comput. Eng. 17(2), 43–49 (2015)Google Scholar
  23. 23.
    Papamitsiou, Z., Economides, A.: Learning analytics and educational data mining in practice: a systematic literature review of empirical evidence. Educ. Technol. Soc. 17(4), 49–64 (2014)Google Scholar
  24. 24.
    Isha, S., Dinesh, K., Mudit, K.: A review of applications of data mining techniques for prediction of students’ performance in higher education. J. Stat. Manage. Syst. 20(4), 713–722 (2017). Scholar
  25. 25.
    Raheela, A., Agathe, M., Syed Abbas, A., Najmi, G.H.: Analyzing undergraduate students’ performance using educational data mining. Comput. Educ. 113, 177–194 (2017)CrossRefGoogle Scholar
  26. 26.
    Anwar, A.Y., Addin, O.: Automatic classification of questions into Bloom’s cognitive levels using support vector machines. In: The International Arab Conference on Information Technology. Naif Arab University for Security Science (NAUSS), Riyadh, Saudi Arabia (2013)Google Scholar
  27. 27.
    Anwar, A.Y., Addin, O., Mohammad, S.E.: Rocchio algorithm-based particle initialization mechanism for effective PSO classification of high dimensional data. Swarm Evol. Comput. 34, 18–32 (2017). Scholar
  28. 28.
    Addin, O., Anwar, A., Y.: Classifications of exam questions using linguistically-motivated features: a case study based on Bloom’s taxonomy. In: The Third International Arab Conference on Quality Assurance in Higher Education (IACQA 2016), pp. 889–896. Khartoum Sudan (2016)Google Scholar
  29. 29.
    Hamalainen, W., Vinni, M.: Comparison of machine learning methods for intelligent tutoring systems. In: ITS 2006 Proceedings of the 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan, pp. 525–534 (2006)Google Scholar
  30. 30.
    Mohamad, S.K., Tasir, Z.: Educational data mining: a review. In: The 9th International Conference on Cognitive Science, pp. 320–324. Procedia - Social and Behavioral Sciences, Kuching, Sarawak, Malaysia (2013)Google Scholar
  31. 31.
    Ronald, D.: The Importance of Having Data-sets. In: Proceedings of the IATUL Conferences, Paper 16 (2006)Google Scholar
  32. 32.
    Anwar, A.Y., Zakaria, T., Addin, O.: Bloom’s Taxonomy–based classification for item bank questions using support vector machines. In: Modern Advances in Intelligent Systems and Tools, vol. 431, pp. 135–140 (2012).
  33. 33.
    Anwar, A.Y., Addin, O.: Automatic classification of questions into Bloom’s cognitive levels using support vector machines. In: The International Arab Conference on Information Technology, pp. 335–342. Naif Arab University for Security Science (NAUSS), Riyadh, Saudi Arabia (2011).
  34. 34.
    Anwar, A.Y., Addin, O., Ahmed A.A.: Educational data mining: a case study of teacher’s classroom questions. In: 13th International Conference on Intelligent Systems Design and Applications (ISDA), pp. 34–41. UPM, Selangor (2013).
  35. 35.
    Koller, D., Sahami, M.: Hierarchically classifying documents using very few words. In: International Conference on Machine Learning (ICML 1997), Nashville, Tennessee, pp. 170–178 (1997)Google Scholar
  36. 36.
    Weigend, A.S., Wiener, E.D., Pedersen, J.O.: Exploiting hierarchy in text categorization. Inf. Retrieval 1(3), 193–216 (1999)CrossRefGoogle Scholar
  37. 37.
    Steven, B., Ewan, K., Edward, L.: Natural Language Processing with Python, 1st edn. O’Reilly Media, USA (2009)zbMATHGoogle Scholar
  38. 38.
    Tsoumakas, G., Katakis, I., Vlahavas, I.: Mining multi-label data. In: Data Mining and Knowledge Discovery Handbook, pp. 667–685. Springer, Heidelberg (2010)Google Scholar
  39. 39.
    Jesse, R., Peter, R., Bernhard, P., Geoff, H.: MEKA: a multi-label/multi-target extension to Weka. J. Mach. Learn. Res. 17(21), 1–5 (2016)MathSciNetzbMATHGoogle Scholar
  40. 40.
    Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann, Elsevier, Amsterdam (2005)zbMATHGoogle Scholar
  41. 41.
    Read, J., Pfahringer, B., Holmes, G.: Multi-label classification using ensembles of pruned sets. In: 8th IEEE International Conference on Data Mining, Pisa, Italy, pp. 995–1000. IEEE Computer Society (2008)Google Scholar
  42. 42.
    Sajnani, H., Javanmardi, S., McDonald, D.W., Lopes, C.V.: Multi-label classification of short text: a study on wikipedia barnstars. In: Analyzing Microtext: the Proceeding of the 2011 AAAI Workshop (2011)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Addin Osman
    • 1
  • Anwar Ali Yahya
    • 1
    • 2
  • Mohammed Basit Kamal
    • 1
  1. 1.College of Computer Science and Information SystemsNajran UniversityNajranSaudi Arabia
  2. 2.Faculty of Computer Science and Information SystemsThamar UniversityThamarYemen

Personalised recommendations