TechCheck: Development and Validation of an Unplugged Assessment of Computational Thinking in Early Childhood Education

Abstract

There is a need for developmentally appropriate Computational Thinking (CT) assessments that can be implemented in early childhood classrooms. We developed a new instrument called TechCheck for assessing CT skills in young children that does not require prior knowledge of computer programming. TechCheck is based on developmentally appropriate CT concepts and uses a multiple-choice “unplugged” format that allows it to be administered to whole classes or online settings in under 15 min. This design allows assessment of a broad range of abilities and avoids conflating coding with CT skills. We validated the instrument in a cohort of 5–9-year-old students (N = 768) participating in a research study involving a robotics coding curriculum. TechCheck showed good reliability and validity according to measures of classical test theory and item response theory. Discrimination between skill levels was adequate. Difficulty was suitable for first graders and low for second graders. The instrument showed differences in performance related to race/ethnicity. TechCheck scores correlated moderately with a previously validated CT assessment tool (TACTIC-KIBO). Overall, TechCheck has good psychometric properties, is easy to administer and score, and discriminates between children of different CT abilities. Implications, limitations, and directions for future work are discussed.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  1. Aho, A. V. (2012). Computation and computational thinking. The Computer Journal, 55(7), 832–835. https://doi.org/10.1093/comjnl/bxs074.

    Article  Google Scholar 

  2. Barr, V., & Stephenson, C. (2011). Bringing computational thinking to K-12: what is involved and what is the role of the computer science education community? Inroads, 2(1), 48–54. https://doi.org/10.1145/1929887.1929905.

    Article  Google Scholar 

  3. Barron, B., Cayton-Hodges, G., Bofferding, L., Copple, C., Darling-Hammond, L., & Levine, M. (2011). Take a giant step: a blueprint for teaching children in a digital age. New York: The Joan Ganz Cooney Center at Sesame Workshop Retrieved from https://joanganzcooneycenter.org.

    Google Scholar 

  4. Bell, T., & Vahrenhold, J. (2018). CS unplugged—how is it used, and does it work?. In Adventures between lower bounds and higher altitudes (pp. 497–521). Springer, Cham. https://doi.org/10.1007/978-3-319-98355-4_29.

  5. Bers, M. U. (2010). The TangibleK robotics program: applied computational thinking for young children. Early Childhood Research and Practice, 12(2) Retrieved from http://ecrp.uiuc.edu/v12n2/bers.html/.

  6. Bers, M. U. (2018). Coding as a playground: programming and computational thinking in the early childhood classroom. Routledge. https://doi.org/10.4324/9781315398945.

  7. Bers, M. U., & Sullivan, A. (2019). Computer science education in early childhood: the case of ScratchJr. Journal of Information Technology Education: Innovations in Practice, 18, 113–138. https://doi.org/10.28945/4437.

    Article  Google Scholar 

  8. Bers, M. U., Flannery, L., Kazakoff, E. R., & Sullivan, A. (2014). Computational thinking and tinkering: exploration of an early childhood robotics curriculum. Computers in Education, 72, 145–157. https://doi.org/10.1016/j.compedu.2013.10.020.

    Article  Google Scholar 

  9. Botički, I., Kovačević, P., Pivalica, D., & Seow, P. (2018). Identifying patterns in computational thinking problem solving in early primary education. Proceedings of the 26th International Conference on Computers in Education. Retrieved from https://www.bib.irb.hr/950389?rad=950389

  10. Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. In Proceedings of the 2012 annual meeting of the American educational research association, Vancouver, Canada (Vol. 1, p. 25).

  11. Cappelleri, J. C., Lundy, J. J., & Hays, R. D. (2014). Overview of classical test theory and item response theory for the quantitative assessment of items in developing patient-reported outcomes measures. Clinical Therapeutics, 36(5), 648–666. https://doi.org/10.1016/j.clinthera.2014.04.006.

    Article  Google Scholar 

  12. Chen, G., Shen, J., Barth-Cohen, L., Jiang, S., Huang, X., & Eltoukhy, M. (2017). Assessing elementary students’ computational thinking in everyday reasoning and robotics programming. Computers in Education, 109, 162–175. https://doi.org/10.1016/j.compedu.2017.03.001.

    Article  Google Scholar 

  13. Code.org (2019). Retrieved from https://code.org/

  14. Core Team, R. (2019). R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing Retrieved from https://www.R-project.org/.

    Google Scholar 

  15. Computer Science Teachers Association (CSTA) Standards Task Force CSTA K-12 computer science standards (2011), p. 9 Retrieved from: http://c.ymcdn.com/sites/www.csteachers.org/resource/resmgr/Docs/Standards/CSTA_K-12_CSS.pdf

  16. Cuny, J., Snyder, L., & Wing, J.M. (2010). Demystifying computational thinking for non-computer scientists. Unpublished manuscript in progress, referenced in http://www.cs.cmu.edu/~CompThink/resources/TheLinkWing.pdf

  17. Dagiene, V., & Stupurienė, G. (2016). Bebras--a sustainable community building model for the concept based learning of informatics and computational thinking. Informatics in education, 15(1), 25–44. https://doi.org/10.15388/infedu.2016.02.

    Article  Google Scholar 

  18. Dienes, Z. (2014). Using Bayes to get the most out of non-significant results. Frontiers in Psychology, 5, 781. https://doi.org/10.3389/fpsyg.2014.00781.

    Article  Google Scholar 

  19. Embretson, S. E., & Reise, S. P. (2000). Multivariate applications books series. Item response theory for psychologists. Mahwah, NJ, US: Lawrence Erlbaum Associates Publishers. Retrieved from https://psycnet.apa.org/record/2000-03918-000

  20. Fayer, S., Lacey, A., & Watson, A. (2017). BLS spotlight on statistics: STEM occupations-past, present, and future. Washington, D.C.: U.S. Department of Labor, Bureau of Labor Statistics. Retrieved from https://www.bls.gov.

  21. Fraillon, J., Ainley, J., Schulz, W., Duckworth, D., & Friedman, T. (2018). International Computer and Information Literacy Study: ICILS 2018: technical report. Retrieved from https://www.springer.com/gp/book/9783030193881

  22. Gamer, M., Lemon, J., Fellows, I. & Singh, P. (2019) Package ‘irr’. Various coefficients of interrater reliability and agreement.Retrieved from https://CRAN.R-project.org/package=irr

  23. Grover, S., & Pea, R. (2013). Computational thinking in K–12: a review of the state of the field. Educational Research, 42(1), 38–43. https://doi.org/10.3102/0013189X12463051.

    Article  Google Scholar 

  24. Hinton, P., Brownlow, C., Mcmurray, I., & Cozens, B. (2004). SPSS explained. Abingdon-on-Thames: Taylor & Francis. https://doi.org/10.4324/9780203642597.

    Google Scholar 

  25. Horn, M. (2012). TopCode: Tangible Object Placement Codes. Retrieved from: http://users.eecs.northwestern.edu/~mhorn/topcodes.

  26. ISTE. (2015). CT leadership toolkit. Retrieved from http://www.iste.org/docs/ct-documents/ct-leadershipttoolkit.pdf?sfvrsn=4

  27. Jeffreys, H. (1961). Theory of probability (3rd ed.). Oxford: Oxford University Press.

    Google Scholar 

  28. K-12 Computer Science Framework Steering Committee. (2016). K–12 computer science framework. Retrieved from https://k12cs.org.

  29. Kalelioğlu, F., Gülbahar, Y., & Kukul, V. (2016). A framework for computational thinking based on a systematic research review. Retrieved from https://www.researchgate.net/publication/303943002_A_Framework_for_Computational_Thinking_Based_on_a_Systematic_Research_Review

  30. Kingsbury, G. G., & Weiss, D. J. (1983). A comparison of IRT-based adaptive mastery testing and a sequential mastery testing procedure. In New horizons in testing (pp. 257-283). Academic Press. https://doi.org/10.1016/B978-0-12-742780-5.50024-X.

  31. Lee, I., Martin, F., Denner, J., Coulter, B., Allan, W., Erickson, J., Malyn-Smith, J., & Werner, L. (2011). Computational thinking for youth in practice. ACM Inroads, 2(1), 32–37. https://doi.org/10.1145/1929887.1929902.

    Article  Google Scholar 

  32. Lu, J. J., & Fletcher, G. H. (2009). Thinking about computational thinking. In ACM SIGCSE Bulletin (Vol. 41, No. 1, pp. 260-264). ACM. https://doi.org/10.1145/1539024.1508959.

  33. Marinus, E., Powell, Z., Thornton, R., McArthur, G., & Crain, S. (2018). Unravelling the cognition of coding in 3-to-6-year olds: the development of an assessment tool and the relation between coding ability and cognitive compiling of syntax in natural language. Proceedings of the 2018 ACM Conference on International Computing Education Research - ICER ’18, 133–141. https://doi.org/10.1145/3230977.3230984.

  34. Mioduser, D., & Levy, S. T. (2010). Making sense by building sense: kindergarten children’s construction and understanding of adaptive robot behaviors. International Journal of Computers for Mathematical Learning, 15(2), 99–127. https://doi.org/10.1007/s10758-010-9163-9.

    Article  Google Scholar 

  35. Moore, T. J., Brophy, S. P., Tank, K. M., Lopez, R. D., Johnston, A. C., Hynes, M. M., & Gajdzik, E. (2020). Multiple representations in computational thinking tasks: a clinical study of second-grade students. Journal of Science Education and Technology, 29(1), 19–34. https://doi.org/10.1007/s10956-020-09812-0.

    Article  Google Scholar 

  36. Morey, R. D., & Rouder, J. N. (2015). BayesFactor 0.9. 12-2. Comprehensive R Archive Network Retrieved from https://cran.r-project.org/web/packages/BayesFactor/index.html.

  37. Papert, S. (1980). Mindstorms: children, computers, and powerful ideas. New York: Basic Books. Retrieved from https://dl.acm.org/citation.cfm?id=1095592.

  38. Ramsay, M. C., & Reynolds, C. R. (2000). Development of a scientific test: a practical guide. Handbook of psychological assessment, 21–42. https://doi.org/10.1016/B978-008043645-6/50080-X.

  39. Relkin, E. (2018). Assessing young children’s computational thinking abilities (Master’s thesis). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 10813994).

  40. Relkin, E., & Bers, M. U. (2019). Designing an assessment of computational thinking abilities for young children. In L. E. Cohen & S. Waite-Stupiansky (Eds.), STEM for early childhood learners: how science, technology, engineering and mathematics strengthen learning. New York: Routledge. https://doi.org/10.4324/9780429453755-5.

    Google Scholar 

  41. Resnick, M. (2007). All I really need to know (about creative thinking) I learned (by studying how children learn) in kindergarten in Proceedings of the 6th Conference on Creativity & Cognition (CC ‘07), pp. 1–6, ACM. https://doi.org/10.1145/1254960.1254961.

  42. Rizopoulos, D. (2006). ltm: an R package for latent variable modelling and item response theory analyses. Journal of Statistical Software, 17(5), 1–25. https://doi.org/10.18637/jss.v017.i05.

    Article  Google Scholar 

  43. Rodriguez, B., Rader, C., & Camp, T. (2016). Using student performance to assess CS unplugged activities in a classroom environment. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (pp. 95-100). ACM. https://doi.org/10.1145/2899415.2899465.

  44. Román-González, M., Pérez-González, J.-C., & Jiménez-Fernández, C. (2017). Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test. Computers in Human Behavior, 72, 678–691. https://doi.org/10.1016/j.chb.2016.08.047.

    Article  Google Scholar 

  45. Román-González, M., Pérez-González, J. C., Moreno-León, J., & Robles, G. (2018). Can computational talent be detected? Predictive validity of the Computational Thinking Test. International Journal of Child-Computer Interaction, 18, 47–58. https://doi.org/10.1016/j.ijcci.2018.06.004.

    Article  Google Scholar 

  46. Román-González, M., Moreno-León, J., & Robles, G. (2019). Combining assessment tools for a comprehensive evaluation of computational thinking interventions. In Computational thinking education (pp. 79–98). Springer, Singapore. Retrieved from https://link.springer.com/chapter/10.1007/978-981-13-6528-7_6.

  47. RStudio Team. (2018). RStudio: integrated development for R. Boston: Studio, Inc. Retrieved from http://www.rstudio.com/.

    Google Scholar 

  48. Selby, C. C., & Woollard, J. (2013). Computational thinking: the developing definition. Paper Presented at the 18th annual conference on innovation and Technology in Computer Science Education, Canterbury. Retreived from https://eprints.soton.ac.uk/356481/.

  49. Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational Research Review, 22, 142–158. https://doi.org/10.1016/j.edurev.2017.09.003.

    Article  Google Scholar 

  50. Sullivan, A., & Bers, M. U. (2016). Girls, boys, and bots: gender differences in young children’s performance on robotics and programming tasks. Journal of Information Technology Education: Innovations in Practice, 15, 145–165. https://doi.org/10.28945/3547.

    Article  Google Scholar 

  51. Tang, X., Yin, Y., Lin, Q., Hadad, R., & Zhai, X. (2020). Assessing computational thinking: a systematic review of empirical studies. Computers in Education, 148, 103798. https://doi.org/10.1016/j.compedu.2019.103798.

    Article  Google Scholar 

  52. U.S. Department of Education, Office of Educational Technology (2017). Reimagining the role of technology in education: 2017 National Education Technology Plan update. Retrieved from https://tech.ed.gov/teacherprep.

  53. Vizner M. Z. (2017). Big robots for little kids: investigating the role of Sale in early childhood robotics kits (Master’s thesis). Available from ProQuest Dissertations and Theses database. (UMI No. 10622097).

  54. Wang, D., Wang, T., & Liu, Z. (2014). A tangible programming tool for children to cultivate computational thinking [research article]. https://doi.org/10.1155/2014/428080.

  55. Werner, L., Denner, J., Campe, S., & Kawamoto, D. C. (2012). The fairy performance assessment: measuring computational thinking in middle school. Proceedings of the 43rd ACM Technical Symposium on Computer Science Education, 215–220. https://doi.org/10.1145/2157136.2157200.

  56. Werner, L., Denner, J., & Campe, S. (2014). Using computer game programming to teach computational thinking skills. Learning, Education And Games, 37. Retrieved from https://dl.acm.org/citation.cfm?id=2811150.

  57. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E. J. (2011). Statistical evidence in experimental psychology: an empirical comparison using 855 t tests. Perspectives on Psychological Science, 6(3), 291–298. https://doi.org/10.1177/1745691611406923.

    Article  Google Scholar 

  58. White House. (2016). Educate to innovate. Retrieved from: https://www.whitehouse.gov/issues/education/k-12/educate-innovate.

  59. Wing, J. M. (2006). Computational thinking. CACM Viewpoint, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215.

    Article  Google Scholar 

  60. Wing, J. M. (2008). Computational thinking and thinking about computing. Philosophical transactions of the royal society of London A: mathematical, physical and engineering sciences, 366(1881), 3717–3725. https://doi.org/10.1098/rsta.2008.0118.

    Article  Google Scholar 

  61. Wing, J. (2011). Research notebook: computational thinking—What and why? The Link Magazine, Spring. Carnegie Mellon University, Pittsburgh. Retrieved from: https://www.cs.cmu.edu/link/research- notebookcomputational-thinking-what-and-why.

  62. Yadav, A., Good, J., Voogt, J., & Fisser, P. (2017). Computational thinking as an emerging competence domain. In Technical and vocational education and training (Vol. 23, pp. 1051–1067). https://doi.org/10.1007/978-3-319-41713-4_49.

  63. Zhang, L., & Nouri, J. (2019). A systematic review of learning computational thinking through Scratch in K-9. Computers in Education, 141, 103607. https://doi.org/10.1016/j.compedu.2019.103607.

    Article  Google Scholar 

Download references

Acknowledgments

We sincerely appreciate our Project Coordinator, Angela de Mik, who made this work possible. We would also like to thank all of the educators, parents, and students who participated in this project, as well as members of the DevTech research group at Tufts University.

Funding

This work was generously supported by the Department of Defense Education Activity (DoDEA) grant entitled “Operation: Break the Code for College and Career Readiness.”

Author information

Affiliations

Authors

Corresponding author

Correspondence to Emily Relkin.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the Tufts University Social, Behavioral & Educational IRB protocol no. 1810044.

Informed Consent

Informed consent was obtained from the educators and parents/guardians of participating students. The students gave their assent for inclusion.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1. The six CT domains covered in TechCheck along with examples of the different tasks used to probe those domains

figurea
figureb

Appendix 2. Difficulty and discrimination indexes for the TechCheck assessment

Question Difficulty index Discrimination index Question Difficulty index Discrimination index
1 − 2.62 1.41 9 − 1.00 0.73
2 − 2.32 1.12 10 − 1.61 1.02
3 − 2.35 1.29 11 − 1.19 1.30
4 − 1.67 1.35 12 − 0.22 1.20
5 − 2.63 0.83 13 − 0.094 1.08
6 0.71 0.98 14 − 1.05 0.65
7 − 1.76 1.06 15 .70 0.525
8 − 1.63 0.88  

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Relkin, E., de Ruiter, L. & Bers, M.U. TechCheck: Development and Validation of an Unplugged Assessment of Computational Thinking in Early Childhood Education. J Sci Educ Technol 29, 482–498 (2020). https://doi.org/10.1007/s10956-020-09831-x

Download citation

Keywords

  • Computational thinking
  • Assessment
  • Unplugged
  • Educational technology
  • Elementary education