Interacting with Intelligent Digital Twins

  • Alexie Dingli
  • Foaad HaddodEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11584)


This paper details the Human Computer Interaction (HCI) components of an Intelligent Digital Twin (IDT) system for a semiconductor manufacturing company using gamming visual effects, 3D Computer Aided Design (CAD) and sound effects. The designed digital twin (DT) system will allow users to detect any irregularities (such as equipment failures and defects) in the manufacturing processes, in a timely manner and test some changes without an actual touching of the physical components. The project aims to design an IDT system that enhances the user’s interaction by visualizing big data in such a way which can be easily understood and processed. Thus, allowing the user to intervene and control the entire production processes from the virtual console. The designed IDT system will enhance the UX of the system through the use of new interaction methodologies. The user will have full control of the data flow which is flowing from a data lake through for example, a gazing, a gesturing and a voice recognition interface which will provide contextual information based upon the user’s viewpoint. The current phase of the project investigates the use of Cave VR technology to improve the immersion and the interaction between the user and the virtual system. We seek to develop a virtual environment that makes the users feel they are naturally interacting in a visually-immersed environment irrespective of where they are located. By enhancing the interaction of the user with these new technologies, we will provide a better UX that would create an efficient system which reduces the overall costs of managing the plant and concretely realize the aspirations of Industry 4.0.


Human Computer Interaction Intelligent Digital Twin User experience Semiconductor manufacturing Interaction 


  1. Balta, E.C., Tilbury, D.M., Barton, K.: A centralized framework for system-level control and management of additive manufacturing fleets. In: 2018 IEEE 14th International Conference on Automation Science and Engineering (CASE), pp. 1071–1078. IEEE, August 2018Google Scholar
  2. Borodulin, K., Radchenko, G., Shestakov, A., Sokolinsky, L., Tchernykh, A., Prodan, R.: Towards digital twins cloud platform: microservices and computational workflows to rule a smart factory. In: Proceedings of the 10th International Conference on Utility and Cloud Computing, pp. 209–210. ACM, December 2017Google Scholar
  3. Boschert, S., Rosen, R.: Digital twin—the simulation aspect. In: Hehenberger, P., Bradley, D. (eds.) Mechatronic Futures, pp. 59–74. Springer, Cham (2016). Scholar
  4. Brenner, B., Hummel, V.: Digital twin as enabler for an innovative digital shopfloor management system in the ESB Logistics Learning Factory at Reutlingen-University. Procedia Manuf. 9, 198–205 (2017)CrossRefGoogle Scholar
  5. Desmet, P.M., Porcelijn, R., Van Dijk, M.B.: Emotional design; application of a research-based design approach. Knowl. Technol. Policy 20(3), 141–155 (2007)CrossRefGoogle Scholar
  6. Dix, A.: Human-computer interaction. In: Liu, L., Özsu, M.T. (eds.) Encyclopedia of Database Systems, pp. 1327–1331. Springer, Boston (2009). Scholar
  7. El Saddik, A.: Digital twins: the convergence of multimedia technologies. IEEE MultiMed. 25(2), 87–92 (2018)CrossRefGoogle Scholar
  8. Grieves, M.: Digital twin: manufacturing excellence through virtual factory replication. White paper (2014)Google Scholar
  9. Grieves, M., Vickers, J.: Digital twin: mitigating unpredictable, undesirable emergent behavior in complex systems. In: Kahlen, F.-J., Flumerfelt, S., Alves, A. (eds.) Transdisciplinary Perspectives on Complex Systems, pp. 85–113. Springer, Cham (2017). Scholar
  10. Kefi, M., Hoang, T.N., Richard, P., Verhulst, E.: An evaluation of multimodal interaction techniques for 3D layout constraint solver in a desktop-based virtual environment. Virtual Reality 22, 1–13 (2018)CrossRefGoogle Scholar
  11. Khakifirooz, M., Chien, C.F., Chen, Y.J.: Bayesian inference for mining semiconductor manufacturing big data for yield enhancement and smart production to empower industry 4.0. Appl. Soft Comput. 68, 990–999 (2018)CrossRefGoogle Scholar
  12. Kuhn, D., Ahrens, J.: Smart manufacturing for continuous, high-technology glass production. In: 77th Conference on Glass Problems: A Collection of Papers Presented at the 77th Conference on Glass Problems, Greater Columbus Convention Center, Columbus, OH, 7–9 November 2016, vol. 612, p. 45. Wiley, May 2017Google Scholar
  13. Kuts, V., Modoni, G.E., Terkaj, W., Tähemaa, T., Sacco, M., Otto, T.: Exploiting factory telemetry to support virtual reality simulation in robotics cell. In: De Paolis, L.T., Bourdot, P., Mongelli, A. (eds.) AVR 2017. LNCS, vol. 10324, pp. 212–221. Springer, Cham (2017). Scholar
  14. Lin, W.D., Low, Y.H., Chong, Y.T., Teo, C.L.: Integrated cyber physical simulation modelling environment for manufacturing 4.0. In: 2018 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), pp. 1861–1865, December 2018Google Scholar
  15. Olsson, T., Lagerstam, E., Kärkkäinen, T., Väänänen-Vainio-Mattila, K.: Expected user experience of mobile augmented reality services: a user study in the context of shopping centres. Pers. Ubiquit. Comput. 17(2), 287–304 (2013)CrossRefGoogle Scholar
  16. Pai, Y.S., Outram, B., Vontin, N., Kunze, K.: Transparent reality: using eye gaze focus depth as interaction modality. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pp. 171–172. ACM, October 2016Google Scholar
  17. Qi, Q., Tao, F.: Digital twin and big data towards smart manufacturing and industry 4.0: 360 degree comparison. IEEE Access 6, 3585–3593 (2018)CrossRefGoogle Scholar
  18. Wang, J., Ma, Y., Zhang, L., Gao, R.X., Wu, D.: Deep learning for smart manufacturing: methods and applications. J. Manuf. Syst. 48, 144–156 (2018)CrossRefGoogle Scholar
  19. Rautaray, S.S., Agrawal, A.: Vision based hand gesture recognition for human computer interaction: a survey. Artif. Intell. Rev. 43(1), 1–54 (2015)CrossRefGoogle Scholar
  20. Rebelo, F., Noriega, P., Duarte, E., Soares, M.: Using virtual reality to assess user experience. Hum. Factors 54(6), 964–982 (2012)CrossRefGoogle Scholar
  21. Ritsos, P.D., Ritsos, D.P., Gougoulis, A.S.: Standards for augmented reality: a user experience perspective. In: International AR Standards Meeting, pp. 1–9, February 2011Google Scholar
  22. Schleich, B., Anwer, N., Mathieu, L., Wartzack, S.: Shaping the digital twin for design and production engineering. CIRP Ann. 66(1), 141–144 (2017)CrossRefGoogle Scholar
  23. Szalma, J.L.: On the application of motivation theory to human factors/ergonomics: motivational design principles for human–technology interaction. Hum. Factors 56(8), 1453–1471 (2014)CrossRefGoogle Scholar
  24. Tao, F., Qi, Q.: New IT driven service-oriented smart manufacturing: framework and characteristics. IEEE Trans. Syst. Man Cybern.: Syst. (2017)Google Scholar
  25. Tao, F., et al.: Digital twin-driven product design framework. Int. J. Prod. Res. 1–19 (2018)Google Scholar
  26. Tao, F., Cheng, J., Qi, Q., Zhang, M., Zhang, H., Sui, F.: Digital twin-driven product design, manufacturing and service with big data. Int. J. Adv. Manuf. Technol. 94(9–12), 3563–3576 (2018)CrossRefGoogle Scholar
  27. Tao, F., Cheng, J., Qi, Q., Zhang, M., Zhang, H., Sui, F.: Digital twin-driven product design, manufacturing and service with big data. Int. J. Adv. Manuf. Technol. 94, 1–14 (2017)Google Scholar
  28. Tuegel, E.J., Ingraffea, A.R., Eason, T.G., Spottswood, S.M.: Reengineering aircraft structural life prediction using a digital twin. Int. J. Aerosp. Eng. 2011 (2011)CrossRefGoogle Scholar
  29. Vélaz, Y., Arce, J.R., Gutiérrez, T., Lozano-Rodero, A., Suescun, A.: The influence of interaction technology on the learning of assembly tasks using virtual reality. J. Comput. Inf. Sci. Eng. 14(4), 041007 (2014)CrossRefGoogle Scholar
  30. Yao, X., Zhou, J., Zhang, J., Boër, C.R.: From intelligent manufacturing to smart manufacturing for Industry 4.0 driven by next generation artificial intelligence and further on. In: 2017 5th International Conference on Enterprise Systems (ES), pp. 311–318. IEEE, September 2017Google Scholar
  31. Yun, S., Park, J.H. and Kim, W.T.: Data-centric middleware based digital twin platform for dependable cyber-physical systems. In: 2017 Ninth International Conference on Ubiquitous and Future Networks (ICUFN), pp. 922–926. IEEE, July 2017Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of MaltaMsidaMalta

Personalised recommendations