NNBlocks: a Blockly framework for AI computing

The Journal of Supercomputing (2021)Cite this article

Abstract

Deep learning compiler tool, Tensor Virtual Machine (TVM), has excellent deployment, compilation, and optimization capabilities supported by the industry following the vigorous growth in neural networks (NN). It has a unified intermediate representation (IR) format that can provide efficient compilation and portability. However, its high operational complexity requires considerable effort in development. For beginners with programming backgrounds, a new and easy-to-use design approach is needed. This paper proposes a visual concept approach that can execute artificial intelligence (AI) computing using block-based tools with AI knowledge. This research also develops a web-based NNBlocks framework that uses this approach to integrate with TVM. We conduct experiments to evaluate this approach: (1) interviewees assessed intuition through operating. (2) Interviewees answered a Usability Metric for User Experience (UMUX) to evaluate usability. (3) Interviewees answered the significance of the theme survey assessment. (4) The impact on the system was evaluated through experiments. The results indicate that interviewees respond positively to the intuitiveness of the framework. The usability evaluation of UMUX meets expectations. The theme survey shows that the framework is significant for AI learning. The experiments of the impact indicate that the framework will not burden the system.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

US$ 39.95

Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Notes

  1. 1.

    https://en.wikipedia.org/wiki/Comparison_of_deep-learning_software.

  2. 2.

    https://onnx.ai.

  3. 3.

    https://developers.google.com/blockly.

  4. 4.

    https://www.tensorflow.org/tensorboard.

  5. 5.

    https://github.com/lutzroeder/netron.

  6. 6.

    https://nodered.org.

  7. 7.

    https://www.lego.com/en-us/themes/mindstorms.

  8. 8.

    https://github.com/BlocklyDuino/BlocklyDuino.

  9. 9.

    https://appinventor.mit.edu.

  10. 10.

    https://blockly-demo.appspot.com/static/demos/blockfactory/index.html.

  11. 11.

    https://github.com/onnx/models.

  12. 12.

    https://mxnet.apache.org/api/python/docs/api/gluon/model_zoo/index.html.

References

  1. 1.

    Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, Kudlur M, Levenberg J, Monga R, Moore S, Murray DG, Steiner B, Tucker P, Vasudevan V, Warden P, Wicke M, Yu Y, Zheng X (2016) TensorFlow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX conference on operating systems design and implementation (OSDI ’16), USENIX Association, Savannah, GA, USA, pp 265–283. https://doi.org/10.5555/3026877.3026899

  2. 2.

    Aivaloglou E, Hermans F (2016) How kids code and how we know: an exploratory study on the scratch repository. In: Proceedings of the 2016 ACM conference on international computing education research (ICER ’16), ACM, Melbourne, VIC, Australia, pp 53–61. https://doi.org/10.1145/2960310.2960325

  3. 3.

    Alturayeif N, Alturaief N, Alhathloul Z (2020) DeepScratch: scratch programming language extension for deep learning education. Int J Adv Comput Sci Appl 11(7):642–650. https://doi.org/10.14569/IJACSA.2020.0110777

    Article  Google Scholar 

  4. 4.

    Bajzek M, Bort H, Hunpatin O, Mivshek L, Much T, O’Hare C (2015) MUzECS: embedded blocks for exploring computer science. In: Proceedings of the IEEE blocks and beyond workshop (B&B ’15), IEEE, Atlanta, GA, USA, pp 127–132. https://doi.org/10.1109/BLOCKS.2015.7369021

  5. 5.

    Bangor A, Kortum PT, Miller JT (2008) An empirical evaluation of the system usability scale. Int J Hum Comput Interact 24(6):574–594. https://doi.org/10.1080/10447310802205776

    Article  Google Scholar 

  6. 6.

    Bastien F, Lamblin P, Pascanu R, Bergstra J, Goodfellow I, Bergeron A, Bouchard N, Warde-Farley D, Bengio Y (2012) Theano: new features and speed improvements. arXiv:cs.SC/1211.5590

  7. 7.

    Benayache A, Bilami A, Barkat S, Lorenz P, Taleb H (2019) MsM: a microservice middleware for smart WSN-based IoT application. J Netw Comput Appl 144:138–154. https://doi.org/10.1016/j.jnca.2019.06.015

    Article  Google Scholar 

  8. 8.

    Brooke J (1996) SUS: a quick and dirty usability scale. In: Usability evaluation in industry, pp 189–194. https://doi.org/10.1201/9781498710411

  9. 9.

    Chen T, Li M, Li Y, Lin M, Wang N, Wang M, Xiao T, Xu B, Zhang C, Zhang Z (2015) MXNet: a flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:cs.DC/1512.01274

  10. 10.

    Chen T, Moreau T, Jiang Z, Zheng L, Yan E, Cowan M, Shen H, Wang L, Hu Y, Ceze L, Guestrin C, Krishnamurthy A (2018) TVM: An automated end-to-end optimizing compiler for deep learning. In: Proceedings of the 13th USENIX conference on operating systems design and implementation (OSDI ’18), USENIX Association, Carlsbad, CA, USA, pp 579–594. https://doi.org/10.5555/3291168.3291211

  11. 11.

    Chen Y, Luca GD (2016) VIPLE: Visual IoT/Robotics programming language environment for computer science education. In: Proceedings of the IEEE international parallel and distributed processing symposium workshops (IPDPSW ’16), IEEE, Chicago, IL, USA, pp 963–971. https://doi.org/10.1109/IPDPSW.2016.55

  12. 12.

    Cong J, Xiao B (2014) Minimizing computation in convolutional neural networks. In: Artificial neural networks and machine learning—ICANN, vol 8681, Springer, pp 281–290. https://doi.org/10.1007/978-3-319-11179-7_36

  13. 13.

    Finstad K (2010) The usability metric for user experience. Interact Comput 22(5):323–327. https://doi.org/10.1016/j.intcom.2010.04.004

    Article  Google Scholar 

  14. 14.

    Glenn T, Ipsita A, Carithers C, Peppler K, Ramani K (2020) StoryMakAR: Bringing stories to life with an augmented reality & physical prototyping toolkit for youth. In: Proceedings of the 2020 CHI conference on human factors in computing systems (CHI ’20), ACM, Honolulu, HI, USA, pp 1–14. https://doi.org/10.1145/3313831.3376790

  15. 15.

    Holwerda R, Hermans F (2018) A usability analysis of blocks-based programming editors using cognitive dimensions. In: Proceedings of the IEEE symposium on visual languages and human-centric computing (VL/HCC ’18), IEEE, Lisbon, Portugal, pp 217–225. https://doi.org/10.1109/VLHCC.2018.8506483

  16. 16.

    Hwang YS, Lin HH, Pai SH, Tu CH (2019) GPUBlocks: GUI Programming Tool for CUDA and OpenCL. J Signal Process Syst 91(3):235–245. https://doi.org/10.1007/s11265-018-1395-2

    Article  Google Scholar 

  17. 17.

    Jatzlau S, Michaeli T, Seegerer S, Romeike R (2019) It’s not magic after all—machine learning in Snap! using reinforcement learning. In: Proceedings of the IEEE blocks and beyond workshop (B&B ’19), IEEE, Memphis, TN, USA, pp 37–41. https://doi.org/10.1109/BB48857.2019.8941208

  18. 18.

    Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on multimedia (MM ’14), ACM, Orlando, Florida, USA, pp 675–678. https://doi.org/10.1145/2647868.2654889

  19. 19.

    Johnson C, Bui P (2015) Blocks in, blocks out: a language for 3D Models. In: Proceedings of the IEEE blocks and beyond workshop (B&B ’15), IEEE, Atlanta, GA, USA, pp 77–82. https://doi.org/10.1109/BLOCKS.2015.7369007

  20. 20.

    Krishnamoorthy SP, Kapila V (2016) Using a visual programming environment and custom robots to learn C programming and K-12 STEM concepts. In: Proceedings of the 6th annual conference on creativity and fabrication in education (FabLearn ’16), ACM, Stanford, CA, USA, pp 41–48. https://doi.org/10.1145/3003397.3003403

  21. 21.

    Maloney J, Resnick M, Rusk N, Silverman B, Eastmond E (2010) The scratch programming language and environment. ACM Trans Comput Educ. https://doi.org/10.1145/1868358.1868363

    Article  Google Scholar 

  22. 22.

    Mao D, Wang F, Wang Y, Hao Z (2019) Visual and user-defined smart contract designing system based on automatic coding. IEEE Access 7:73131–73143. https://doi.org/10.1109/ACCESS.2019.2920776

    Article  Google Scholar 

  23. 23.

    Maquil V, Moll C, Schwartz L, Hermen J (2018) Kniwwelino: A lightweight and WiFi enabled prototyping platform for children. In: Proceedings of the twelfth international conference on tangible, embedded, and embodied interaction (TEI ’18), ACM, Stockholm, Sweden, pp 94–100. https://doi.org/10.1145/3173225.3173275

  24. 24.

    Mattioli A, Paternò F (2020) A visual environment for end-user creation of IoT customization rules with recommendation support. In: Proceedings of the international conference on advanced visual interfaces (AVI ’20), ACM, Salerno, Italy. https://doi.org/10.1145/3399715.3399833

  25. 25.

    Paramasivam V, Huang J, Elliott S, Cakmak M (2017) Computer science outreach with end-user robot-programming tools. In: Proceedings of the 2017 ACM SIGCSE technical symposium on computer science education (SIGCSE ’17), ACM, Seattle, Washington, USA, pp 447–452. https://doi.org/10.1145/3017680.3017796

  26. 26.

    Poole M (2017) Extending the design of a blocks-based python environment to support complex types. In: Proceedings of the IEEE blocks and beyond workshop (B&B ’17), IEEE, Raleigh, NC, USA, pp 1–7. https://doi.org/10.1109/BLOCKS.2017.8120400

  27. 27.

    Protzenko J (2015) Pushing blocks all the way to C++. In: Proceedings of the IEEE blocks and beyond workshop (B&B ’15), IEEE, Atlanta, GA, USA, pp 91–95. https://doi.org/10.1109/BLOCKS.2015.7369010

  28. 28.

    Rao A, Bihani A, Nair M (2018) Milo: A visual programming environment for Data Science Education. In: 2018 IEEE symposium on visual languages and human-centric computing (VL/HCC ’18), IEEE, Lisbon, Portugal, pp 211–215. https://doi.org/10.1109/VLHCC.2018.8506504

  29. 29.

    Rodríguez FJ, Smith RC, Smith A, Boyer KE, Wiebe EN, Mott BW, Lester JC (2019) Toward a responsive interface to support novices in block-based programming. In: Proceedings of the IEEE blocks and beyond workshop (B&B ’19), IEEE, Memphis, TN, USA, pp 9–13. https://doi.org/10.1109/BB48857.2019.8941205

  30. 30.

    Rodríguez-Gil L, García-Zubia J, Orduña P, Villar-Martinez A, López-De-Ipiña D (2019) New approach for conversational agent definition by non-programmers: a visual domain-specific language. IEEE Access 7:5262–5276. https://doi.org/10.1109/ACCESS.2018.2883500

    Article  Google Scholar 

  31. 31.

    Roesch J, Lyubomirsky S, Weber L, Pollock J, Kirisame M, Chen T, Tatlock Z (2018) Relay: A new IR for machine learning frameworks. In: Proceedings of the 2nd ACM SIGPLAN international workshop on machine learning and programming languages (MAPL ’18), ACM, Philadelphia, PA, USA, pp 58–68. https://doi.org/10.1145/3211346.3211348

  32. 32.

    Seraj M, Große CS, Autexier S, Drechsler R (2019) Look what i can do: acquisition of programming skills in the context of living labs. In: Proceedings of the 41th IEEE/ACM international conference on software engineering: software engineering education and training (ICSE-SEET ’19), IEEE, Montreal, QC, Canada, pp 197–207. https://doi.org/10.1109/ICSE-SEET.2019.00029

  33. 33.

    Serna MÁ, Sreenan CJ, Fedor S (2015) A visual programming framework for wireless sensor networks in smart home applications. In: Proceedings of the IEEE tenth international conference on intelligent sensors, sensor networks and information processing (ISSNIP ’15), IEEE, Singapore, Singapore, pp 1–6. https://doi.org/10.1109/ISSNIP.2015.7106946

  34. 34.

    Stewart AE, Keirn ZA, D’Mello SK (2018) Multimodal modeling of coordination and coregulation patterns in speech rate during triadic collaborative problem solving. In: Proceedings of the 20th ACM international conference on multimodal interaction (ICMI ’18), ACM, Boulder, CO, USA, pp 21–30. https://doi.org/10.1145/3242969.3242989

  35. 35.

    Tokui S, Okuta R, Akiba T, Niitani Y, Ogawa T, Saito S, Suzuki S, Uenishi K, Vogel B, Yamazaki Vincent H (2019) Chainer: A deep learning framework for accelerating the research cycle. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining (KDD ’19), ACM, Anchorage, AK, USA, pp 2002–2011. https://doi.org/10.1145/3292500.3330756

  36. 36.

    Tomlein M, Grønbæk K (2017) A visual programming approach based on domain ontologies for configuring industrial IoT installations. In: Proceedings of the seventh international conference on the internet of things IoT ’17, ACM, Linz, Austria, pp 1–9. https://doi.org/10.1145/3131542.3131552

  37. 37.

    Vasilache N, Zinenko O, Theodoridis T, Goyal P, DeVito Z, S Moses W, Verdoolaege S, Adams A, Cohen A (2018) Tensor comprehensions: framework-agnostic high-performance machine learning abstractions. arXiv:cs.PL/1802.04730

  38. 38.

    Weintrop D, Wilensky U (2017) Comparing block-based and text-based programming in high school computer science classrooms. ACM Trans Comput Educ. https://doi.org/10.1145/3089799

    Article  Google Scholar 

  39. 39.

    Weintrop D, Shepherd CD, Francis P, Franklin D (2017) Blockly goes to work: block-based programming for industrial robots. In: Proceedings of the IEEE blocks and beyond workshop (B&B ’17), IEEE, Raleigh, NC, USA, pp 29–36. https://doi.org/10.1109/BLOCKS.2017.8120406

  40. 40.

    Zatarain-Cabada R, Barrón-Estrada ML (2016) Ríos-Félix JM (2017) Affective learning system for algorithmic logic applying gamification. In: Advances in soft computing—MICAI, vol 10062, Springer, pp 536–547. https://doi.org/10.1007/978-3-319-62428-0_44

Download references

Acknowledgements

The work is supported by the Taiwan Ministry of Science and Technology under Grant No.: MOST 107-2221-E-007-005-MY3.

Author information

Affiliations

  1. Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan

    Tai-Liang Chen, Yi-Ru Chen, Meng-Shiun Yu & Jenq-Kuen Lee

Authors
  1. Tai-Liang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yi-Ru Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meng-Shiun Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jenq-Kuen Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jenq-Kuen Lee.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chen, TL., Chen, YR., Yu, MS. et al. NNBlocks: a Blockly framework for AI computing. J Supercomput (2021). https://doi.org/10.1007/s11227-021-03631-9

Download citation

Keywords

  • Blockly
  • Visualization
  • Visual block
  • Scheduling optimization
  • Intermediate representation
  • Neural network