Abstract
Graphical user interface (GUI) is very important to interact with software users. In many studies, therefore, they are trying to convert GUI elements (or widgets) to code or to describe formally its structure by help of domain knowledge or machine learning based algorithms. In this paper, we adopted object detection based on deep neural networks that finds GUI elements by integration of localization and classification. After the successfully detection of GUI components, we will describe the objects as the hierarchical structure and transform those to appropriate codes by synthetic or machine learning algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Parag, T., Bahlmann, C., Shet, V., Singh, M.: A grammar for hierarchical object descriptions in logic programs. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, pp. 33–38 (2012)
Pham, H., Nguyen, T., Vu, P., Nguyenm, T.: Toward mining visual log of software. arXiv e-prints arXiv:1610.08911, October 2016
Beltramelli, T.: pix2code: generating code from a graphical user interface screenshot. arXiv e-prints arXiv:1705.07962, June 2017
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, pp. 580–587 (2014)
Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. arXiv e-prints arXiv:1612.08242v1, December 2016
Tzutalin: LabelImg, Git code (2015). https://github.com/tzutalin/labelImg
YOLO: real-time object detection. https://pjreddie.com/darknet/yolo/
Acknowledgment
This research was supported by Next-Generation Information Computing Development Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT (No. NRF-2017M3C4A7069073).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yun, YS., Jung, J., Eun, S., So, SS., Heo, J. (2019). Detection of GUI Elements on Sketch Images Using Object Detector Based on Deep Neural Networks. In: Hwang, S., Tan, S., Bien, F. (eds) Proceedings of the Sixth International Conference on Green and Human Information Technology. ICGHIT 2018. Lecture Notes in Electrical Engineering, vol 502. Springer, Singapore. https://doi.org/10.1007/978-981-13-0311-1_16
Download citation
DOI: https://doi.org/10.1007/978-981-13-0311-1_16
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-0310-4
Online ISBN: 978-981-13-0311-1
eBook Packages: EngineeringEngineering (R0)