Skip to main content
SpringerLink
Log in

A Knowledge Points and Cognitive Verb Labeling Strategy for Test Questions Based on Crowdsourcing Mode

  • Conference paper
  • First Online:
Knowledge and Systems Sciences (KSS 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1103))

Included in the following conference series:

  • 577 Accesses

Abstract

Crowdsourcing, as an open knowledge production process, has the characteristics of voluntary and collaborative sharing, and has achieved remarkable results in solving many complex practical problems. In order to solve the labeling problem of knowledge points and corresponding cognitive verbs in massive test questions, this paper proposes a labeling strategy based on crowdsourcing mode. Firstly, on the basis of the three modes of crowdsourcing and the context of the problem, a crowdsourcing labeling strategy for knowledge points and cognitive verbs suitable for high school mathematics test questions is proposed. Secondly, experiments are designed and carried out to verify the feasibility of the strategy. Finally, a method of gold standard combined with adapted EM algorithm is put forward to control the quality of crowdsourcing results, and the pricing of such tasks is obtained based on experimental data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Howe, J.: The rise of crowdsourcing. Wired 14(6), 176–183 (2006)

    Google Scholar 

  2. Sindlinger, T.S.: Crowdsourcing: why the power of the crowd is driving the future of business. Am. J. Health Syst. Pharmacy. 67(18), 1565–1566 (2010)

    Article  Google Scholar 

  3. Omar, F.Z., Chris, C.: Crowdsourcing translation: professional quality from non-professionals. In: Meeting of the Association for Computational Linguistics: Human Language Technologies, DBLP (2011)

    Google Scholar 

  4. Abduljabbar, D.A., Omar, N.: Exam questions classification based on Bloom’s taxonomy cognitive level using classifiers combination. J. Theor. Appl. Inf. Technol. 78, 447–455 (2015)

    Google Scholar 

  5. George, D., Jimenez, S., Baquero, J.: Automatic prediction of item difficulty for short-answer questions. In: Computing Colombian Conference. IEEE (2015)

    Google Scholar 

  6. Piech, C., Bassen, J., Huang, J., et al.: Deep knowledge tracing. In: Proceedings of the 28th International Conference on Neural Information Processing Systems. MIT Press (2015)

    Google Scholar 

  7. Milicevic, A.K., Vesin, B., Ivanovic, M., et al.: E-Learning personalization based on hybrid recommendation strategy and learning style identification. Comput. Educ. 56(3), 885–899 (2011)

    Article  Google Scholar 

  8. Bloom, B.S., Engelhart, M.D., Furst, E.J., et al.: Taxonomy of Educational Objectives, Hand-Book I: The Cognitive Domain. David McKay Co Inc., New York (1956)

    Google Scholar 

  9. Midgley, C.: Goals, Goal Structures and Patterns of Adaptive Learning. Routledge, New York (2014)

    Book  Google Scholar 

  10. Hudak, G.M., Kihn, P.: Labeling: Pedagogy and Politics. Routledge, New York (2014)

    Book  Google Scholar 

  11. Wang, Y., Sun, H.: The design of instructional objectives. Foreign Language Teaching and Research Press, Beijing (2017)

    Google Scholar 

  12. Pietro, M., Janis, L.D.: The power of crowds. Science 351(6268), 32–33 (2016)

    Article  Google Scholar 

  13. Pénin, J.: The limits of crowdsourcing inventive activities: what do transaction cost theory and the evolutionary theories of the firm teach us? http://cournot.u-strasbg.fr/users/osi/program/TBH_JP_crowdsouring%202010%20ENG.pdf. Accessed 11 July 2019

  14. Panchal, L.Q., Panchal, J.H.: Modeling the effect of product architecture on mass-collaborative processes. J. Comput. Inf. Sci. Eng. 11, 23–46 (2011)

    Article  Google Scholar 

  15. Hirth, M., Hofeld, T., Tran-Gia, P.: Cheat-detection mechanisms for crowdsourcing. Technical report 474, University of Wurzburg (2010)

    Google Scholar 

  16. Ipeirotis, P. G., Provost, F., Wang, J.: Quality management on Amazon mechanical turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 64–67. ACM (2010)

    Google Scholar 

  17. Raykar, V.C., Yu, S.: Ranking annotators for crowdsourced labeling tasks. In: Advances in Neural Information Processing Systems, pp. 1809–1817 (2011)

    Google Scholar 

  18. Aytaç, K., Ada, T.: Evaluation of mathematics teacher candidates’ the ellipse knowledge according to the revised bloom’s taxonomy. Univers. J. Educ. Res. 5(10), 1782–1794 (2017)

    Article  Google Scholar 

  19. Sharunova, A., Butt, M., Qureshi, A.J.: Transdisciplinary design education for engineering undergraduates: mapping of Bloom’s taxonomy cognitive domain across design stages. Procedia CIRP 70, 313–318 (2018)

    Article  Google Scholar 

  20. Verenna, A.A., Noble, K.A., Pearson, H.E., et al.: Role of comprehension on performance at higher levels of Bloom’s taxonomy: Findings from assessments of healthcare professional students. Anat. Sci. Educ. 11(5), 433–444 (2018)

    Article  Google Scholar 

  21. Arneson, J.B., Offerdahl, E.G.: Visual literacy in Bloom: using Bloom’s taxonomy to support visual learning skills. CBE Life Sci. Educ. 17(1), ar7 (2018)

    Article  Google Scholar 

  22. Radmehr, F., Drake, M.: Revised Bloom’s taxonomy and integral calculus: unpacking the knowledge dimension. Int. J. Math. Educ. 48(8), 1206–1224 (2017)

    Google Scholar 

  23. Amorim, G.F., Balestrassi, P.P., Sawhney, R., et al.: Six Sigma learning evaluation model using Bloom’s taxonomy. Int. J. Lean Six Sigma 9(1), 156–174 (2018)

    Article  Google Scholar 

  24. Jolliffe, F., Ponsford, R.A.: Classification and comparison of mathematics examinations—methods based on Bloom’s taxonomy. Int. J. Math. Educ. Sci. Technol. 20(5), 677–688 (1989)

    Article  Google Scholar 

  25. Kusuma, S.F., Siahaan, D., Yuhana, U.L.: Automatic Indonesia’s questions classification based on bloom’s taxonomy using Natural Language Processing a preliminary study. In: International Conference on Information Technology Systems & Innovation. IEEE (2016)

    Google Scholar 

  26. Diab, S., Sartawi, B.: Classification of questions and learning outcome statements (LOS) into Bloom’s taxonomy (BT) by similarity measurements towards extracting of learning outcome from learning material. Int. J. Manag. Inf. Technol. 9(2), 01–12 (2017)

    Google Scholar 

  27. Guo, C.: Big Data and Protection of Ancient Villages in China. South China University of Technology Press, Guangzhou (2017)

    Google Scholar 

  28. Qiao, C., Hu, X.: Text classification for cognitive domains: a case using lexical, syntactic and semantic features. J. Inf. Sci. 45(4), 516–528 (2019)

    Article  Google Scholar 

  29. Mason, W.A., Watts, D.J.: Financial incentives and the “performance of crowds”. ACM SIGKDD Explor. Newsl. 11(2), 100–108 (2010)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Natural Science Foundation of China [Grant Numbers 71771034, 71421001].

Author information

Authors and Affiliations

  1. Institute of System Engineering, Dalian University of Technology, Dalian, China

    Chonghui Guo & Meng Xu

Authors
  1. Chonghui Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meng Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chonghui Guo .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Jian Chen

  2. Japan Advanced Institute of Science and Technology, Nomi, Japan

    Van Nam Huynh

  3. Duy Tan University, Da Nang, Vietnam

    Gia-Nhu Nguyen

  4. CAS Academy of Mathematics and Systems Sciences, Beijing, China

    Xijin Tang

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Guo, C., Xu, M. (2019). A Knowledge Points and Cognitive Verb Labeling Strategy for Test Questions Based on Crowdsourcing Mode. In: Chen, J., Huynh, V., Nguyen, GN., Tang, X. (eds) Knowledge and Systems Sciences. KSS 2019. Communications in Computer and Information Science, vol 1103. Springer, Singapore. https://doi.org/10.1007/978-981-15-1209-4_9

Download citation

  • DOI: https://doi.org/10.1007/978-981-15-1209-4_9

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-15-1208-7

  • Online ISBN: 978-981-15-1209-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation