Knowledge Capture and Reuse Through Expert’s Activity Monitoring in Engineering Design

  • Harvey RowsonEmail author
  • Matthieu Bricogne
  • Lionel Roucoules
  • Alexandre Durupt
  • Benoit Eynard
Conference paper
Part of the IFIP Advances in Information and Communication Technology book series (IFIPAICT, volume 540)


This paper deals with artificial intelligence driven product engineering support. Many software systems are available to support the product lifecycle, especially during product design, such as CAD, PDM, CAE, SDM, etc. Most product development process is performed using these systems, which through their rich user interfaces allow skilled professionals to express their expertise and knowledge using the tools and functions the software is willing to provide them. At the end of the day, the result of their work is a model, built through a user interface, and stored in a repository. The goal of our research is to reverse engineer the user’s knowledge by analysing his/her actions with the software system, based on the assumption that the process will itself be meta-knowledge driven and that we will focus on engineering software which provide semantically rich user interfaces. The aim of this paper is to investigate the idea of building reusable expert knowledge from actions on engineering software user interfaces. It first outlines existing works from different fields and identifies remaining issues. It then suggest an approach to address these issues and put together an operational system.


Artificial intelligence Knowledge based engineering Engineering design GUI monitoring Computer vision 


  1. Alégroth, E., Feldt, R.: On the long-term use of visual gui testing in industrial practice: a case study. Empir. Softw. Eng. (2017).
  2. Börjesson, E., Feldt, R.: Automated system testing using visual GUI testing tools: a comparative study in industry. In: Proceedings of IEEE 5th International Conference Software Testing, Verification Validation, ICST 2012, pp. 350–359 (2012).
  3. Chang, T.H., Yeh, T., Miller, R.C.: GUI testing using computer vision. In: Proceedings of 28th International Conference Human Factors Computing Systems, pp. 1535–1544 (2010).
  4. Danjou, S., Lupa, N., Koehler, P.: Approach for automated product modeling using knowledge-based design features (2008)Google Scholar
  5. Dekhtiar, J., Durupt, A., Bricogne, M., Eynard, B., Rowson, H., Kiritsis, D.: Deep learning for big data applications in CAD and PLM – research review, opportunities and case study. Comput. Ind. 100, 227–243 (2018). Scholar
  6. Dominic, S., Bügler, M., Borrmann, A.: Knowledge based bridge engineering - artificial intelligence meets building information modeling (2016)Google Scholar
  7. Intharah, T., Turmukhambetov, D., Brostow, G.J.: Help, it looks confusing. In: Proceedings of the 22nd International Conference on Intelligent User Interfaces - IUI 2017, pp. 233–243. ACM Press, New York (2017)Google Scholar
  8. Leo, M., Medioni, G., Trivedi, M., Kanade, T., Farinella, G.M.: Computer vision for assistive technologies. Comput. Vis. Image Underst. 154, 1–15 (2017). Scholar
  9. Memon, A., Banerjee, I., Nagarajan, A.: GUI ripping: reverse engineering of graphical user interfaces for testing (2003)Google Scholar
  10. Moreira, R., Lopes de Matos, M.: Pattern-Based GUI Testing. PQDT - Glob 180 (2014)Google Scholar
  11. Quintana-Amate, S., Bermell-Garcia, P., Tiwari, A.: Transforming expertise into knowledge-based engineering tools: a survey of knowledge sourcing in the context of engineering design. Knowl.-Based Syst. 84, 89–97 (2015). Scholar
  12. Qureshi, I.A., Nadeem, A.: GUI testing techniques: a survey. Int. J. Future Comput. Commun. 142–146 (2013).
  13. Reddy, E.J., Sridhar, C.N.V., Rangadu, V.P.: Knowledge based engineering: notion, approaches and future trends. Am. J. Intell. Syst. 5, 1–17 (2015). Scholar
  14. La, R.G.: Knowledge based engineering: between AI and CAD. Review of a language based technology to support engineering design. Adv. Eng. Inform. 26, 159–179 (2012). Scholar
  15. Sadeghi, S., Dargon, T., Rivest, L., Pernot, J.-P.: Capturing and analysing how designers use CAD software (2016)Google Scholar
  16. Satama, M.: Event capturing tool for model-based GUI test automation (2006)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2018

Authors and Affiliations

  • Harvey Rowson
    • 1
    • 3
    Email author
  • Matthieu Bricogne
    • 1
  • Lionel Roucoules
    • 2
  • Alexandre Durupt
    • 1
  • Benoit Eynard
    • 1
  1. 1.Université de Technologie de CompiègneCompiègneFrance
  2. 2.Ecole Nationale Supérieure d’Arts et MétiersAix-en-ProvenceFrance
  3. 3.DeltaCADLacroix Saint-OuenFrance

Personalised recommendations