Picture-Driven User Interface Development for Applications on Multi-platforms

  • Vinh-Tiep Nguyen
  • Minh-Triet Tran
  • Anh-Duc Duong
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8510)


Graphical user interfaces are usually first sketched out manually as hand drawing pictures and then must be realized by software developers to become prototypes or usable user interfaces. This motivates our proposal of a smart CASE tool that can understand hand drawing sketches of graphical user interfaces, including forms and their navigations, then automatically transform such draft designs into real user interfaces of a prototype or an application. By using the ideas of modeling and model-transformation in model driven engineering, the authors also propose a mechanism to generate graphical user interfaces as forms targeting different platforms. Experimental results show that our sketch recognition to understand hand drawing graphical user interfaces can achieve the accuracy of 97.86% and 95% in recognizing 7 common UI controls and arrows for navigation respectively. Our model transformation engine can generate user interfaces as forms for applications on 3 different platforms of mobile devices, including Windows Phone, Android, and iOS. This approach follows the trend to develop a new generation of smart CASE tools that can understand and interpret conceptual software design models into concrete software elements and components to assist the software development process in a natural way.


picture-driven graphical user interface code generation mobile device multi-platform 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hardesty, L.: Picture-driven computing (January 2010),
  2. 2.
    Lewis, J.P.: Fast Template Matching. In: Vision Interface 1995, pp. 120–123. Candian Image Processing and Pattern Recognition Society, Quebec City (1995)Google Scholar
  3. 3.
    Ouyang, W., Cham, W.K.: Fast algorithm for Walsh Hadamard transform on sliding windows. IEEE Transaction on Pattern Analysis and Machine Intelligence, 165–171 (2010)Google Scholar
  4. 4.
    Hofhauser, A., Steger, C., Navab, N.: Edge-Based Template Matching and Tracking for Perspectively Distorted Planar Objects. In: Bebis, G., et al. (eds.) ISVC 2008, Part I. LNCS, vol. 5358, pp. 35–44. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  5. 5.
    Lowe, D.G.: Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision (IJCV), 91–110 (2004)Google Scholar
  6. 6.
    Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: SURF: Speeded Up Robust Features. Computer Vision and Image Understanding (CVIU), 346–359 (2008)Google Scholar
  7. 7.
    Dalal, N., Triggs, B.: Histograms of Oriented Gradients for Human Detection. In: Conference on Computer Vision and Pattern Recognition, pp. 886–893 (2005)Google Scholar
  8. 8.
    Sun, Z., Liang, S.: Sketch retrieval and relevance feedback with biased SVM classification. Pattern Recognition Letters, 1733–1741 (2008)Google Scholar
  9. 9.
    Su, M.C., Hsio, T.H., Hsieh, Y.Z., Lin, S.C.: A neural-network-based sketch recognition system. In: International Symposium on Intelligent Signal Processing and Communications Systems (ISPACS), pp. 420–423 (2012)Google Scholar
  10. 10.
    Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors. Transactions on Visualization and Computer Graphics, 1624–1636 (2011)Google Scholar
  11. 11.
    Mourouzis, A., Leonidis, A., Foukarakis, M., Antona, M., Maglaveras, N.: A Novel Design Approach for Multi-device Adaptable User Interfaces: Concepts, Methods and Examples. In: Stephanidis, C. (ed.) Universal Access in HCI, Part I, HCII 2011. LNCS, vol. 6765, pp. 400–409. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  12. 12.
    Balagtas-Fernandez, F.T., Hussmann, H.: Model-Driven Development of Mobile Applications. In: International Conference on Automated Software Engineering (ASE 2008), pp. 509–512 (2008)Google Scholar
  13. 13.
    Kherraf, S., Lefebvre, E., Suryn, W.: Transformation From CIM to PIM Using Patterns and Archetypes. In: 19th Australian Conference on Software Engineering (ASWEC 2008), pp. 338–346 (2008)Google Scholar
  14. 14.
    Hammond, T.A.: LADDER: A Perceptually-based Language to Simplify Sketch Recognition User Interface Development, Massachusetts Institute of Technology, Doctor of Philosophy thesis (2007)Google Scholar
  15. 15.
    Diep, C.-K., Tran, Q.-N., Tran, M.-T.: Online model-driven IDE to design GUIs for cross-platform mobile applications. In: The 4th Symposium on Information and Communication Technology, pp. 294–300 (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Vinh-Tiep Nguyen
    • 1
  • Minh-Triet Tran
    • 1
  • Anh-Duc Duong
    • 2
  1. 1.Faculty of Information TechnologyUniversity of Science, VNU-HCMVietnam
  2. 2.Faculty of Software EngineeringUniversity of Information Technology, VNU-HCMVietnam

Personalised recommendations