Sky Writer: Towards an Intelligent Smart-phone Gesture Tracing and Recognition Framework

  • Nicholas Mitri
  • Mariette AwadEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11198)


We present Sky Writer, an intelligent smartphone gesture tracking and recognition framework for free-form gestures. The design leverages anthropomorphic kinematics and device orientation to estimate the trajectory of complex gestures instead of employing traditional acceleration based techniques. Orientation data are transformed, using the kinematic model, to a 3D positional data stream, which is flattened, scaled down, and curve fitted to produce a gesture trace and a set of accompanying features for a support vector machine (SVM) classifier. SVM is the main classifier we adopted but for the sake of comparison, we couple our results with the hidden Markov models (HMM). In this experiment, a dataset of size 1200 is collected from 15 participants that performed 5 instances for each of 16 distinct custom developed gestures after being instructed on how to handle the device. User-dependent, user-independent, and hybrid/mixed learning scenarios are used to evaluate the proposed design. This custom developed gesture set achieved using SVM 96.55%, 96.1%, and 97.75% average recognition rates across all users for the respective learning scenarios.


Support vector machines Gesture recognition Forward kinematics Inverse kinematics Hidden Markov models Machine learning 


  1. Android developers, “Sensor Overview” (2016). Accessed 5 Jan 2016
  2. Amma, C., Georgi, M., Schultz, T.: Airwriting: hands-free mobile text input by spotting and continuous recognition of 3d-space handwriting with inertial sensors. In: 2012 16th International Symposium on Wearable Computers (ISWC), pp. 52–59 (2012). Accessed 18–22 June 2012Google Scholar
  3. Awad, M., Khanna, R.: Efficient Learning Machines: Theories, Concepts, and Applications for Engineers and System Designers. A Press (2015)Google Scholar
  4. Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin classifiers. In: Proceedings of the 5th Annual Workshop on Computational Learning Theory, pp. 144–152 (1992)Google Scholar
  5. Chan, K., Koh, C., Lee, C.S.G.: A 3-D-point-cloud system for human-pose estimation. IEEE Trans. Syst. Man Cybern.: Syst. 44(11), 1486–1497 (2014)CrossRefGoogle Scholar
  6. Cho, S., et al.: Magic wand: a hand-drawn gesture input device in 3-D space with inertial sensors. In: IWFHR-9 2004. Ninth International Workshop on Frontiers in Handwriting Recognition, 2004, pp. 106–111 (2004) 26–29 Oct 2004Google Scholar
  7. Choi, E.S., et al.: Beatbox music phone: gesture-based interactive mobile phone using a tri-axis accelerometer. In: IEEE International Conference on Industrial Technology, 2005. ICIT 2005, pp. 97–102 (2005). Accessed 14–17 Dec 2005Google Scholar
  8. Choi, S., Lee, A.S., Lee, S.Y.: On-line handwritten character recognition with 3D accelerometer. In: International Conference on Information Acquisition, pp. 845–850 (2006). Accessed 20–23 Aug 2006Google Scholar
  9. Craig, J.J.: Introduction to Robotics: Mechanics and Control, 3rd edn. Prentice Hall, Upper Saddle River (2004)Google Scholar
  10. Elmezain, M., et al.: A Hidden Markov Model-based continuous gesture recognition system for hand motion trajectory. In: ICPR 2008. 19th International Conference on Pattern Recognition, 2008, pp. 1–4 (2008). Accessed 8–11 Dec 2008Google Scholar
  11. Fuccella, V., Costagliola, G.: Unistroke Gesture recognition through polyline approximation and alignment. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 3351–3354. ACM (2015)Google Scholar
  12. Fujioka, H., et al.: Constructing and reconstructing characters, words, and sentences by synthesizing writing motions. IEEE Trans. Syst. Man Cybern. Part A: Syst. Hum. 36(4), 661–670 (2006)CrossRefGoogle Scholar
  13. He, Z.: A new feature fusion method for gesture recognition based on 3D accelerometer. In: 2010 Chinese Conference on Pattern Recognition (CCPR), pp. 1–5 (2010). Accessed 21–23 Oct 2010Google Scholar
  14. He, Z.: Accelerometer based gesture recognition using fusion features and SVM. J. Softw. 6(6), 1042–1049 (2011)Google Scholar
  15. Kallio, S., Kela, J., Mantyjarvi, J.: Online gesture recognition system for mobile interaction. In: IEEE International Conference on Systems, Man and Cybernetics, 2003, vol. 3, pp. 2070–2076 (2003). Accessed 5–8 Oct 2003Google Scholar
  16. Kratz, S., Rohs, M., Essl, G.: Combining acceleration and gyroscope data for motion gesture recognition using classifiers with dimensionality constraints. In: Proceedings of the 2013 International Conference on Intelligent user Interfaces, pp. 173–178. ACM (2013)Google Scholar
  17. Liu, J., et al.: uWave: accelerometer-based personalized gesture recognition and its applications. In: PerCom 2009. IEEE International Conference on Pervasive Computing and Communications, 2009, pp. 1–9 (2009). Accessed 9–13 March 2009Google Scholar
  18. Mitri, N., Wilkerson, C., Awad, M.: Recognition of free-form gestures from orientation tracking of a handheld or wearable device (2016).
  19. Prigge, E.: A positioning system with no line-of-sight restrictions for cluttered environments. PhD dissertation, Stanford University, p. 1 (2004)Google Scholar
  20. Kauppila, M., et al.: Accelerometer based gestural control of browser applications. In: Proceedings of UCS, Tokyo (2007)Google Scholar
  21. Khan, M.: Cubic Bezier least square fitting. Matlab Central (2007).
  22. Pylvänäinen, T.: Accelerometer Based gesture recognition using continuous HMMs. In: Marques, Jorge S., Pérez de la Blanca, N., Pina, P. (eds.) IbPRIA 2005. LNCS, vol. 3522, pp. 639–646. Springer, Heidelberg (2005). Scholar
  23. Raffa, G., et al.: Don’t slow me down: bringing energy efficiency to continuous gesture recognition. In: 2010 International Symposium on Wearable Computers (ISWC), pp. 1–8. Accessed 10–13 Oc. 2010Google Scholar
  24. Rico, J., Brewster, S.: Gesture and voice prototyping for early evaluations of social acceptability in multimodal interfaces. In: Proceeding ICMI-MLMI ‘10 International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, Article No. 16Google Scholar
  25. Ruiz, J., Li, Y., Lank, E.: User-defined motion gestures for mobile interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 197–206. ACM (2011)Google Scholar
  26. SmartWatchr: 2015 Smartwatch Specs Comparison Chart. Smartwatch News. (2015).
  27. Saab, A., Mitri, N., Awad, M.: Ham or Spam? A comparative study for some content-based classification algorithms for email filtering. In: 17th IEEE Mediterranean Electrotechnical Conference - Information & Communication Systems, Beirut, Lebanon (2014). Accessed 13–16 April 2014Google Scholar
  28. Timo, P.: Accelerometer based gesture recognition using continuous HMMs. In: Pattern Recognition and Image Analysis Lecture Notes in Computer Science, vol. 3522, pp. 639–646 (2005)Google Scholar
  29. Vapnik, V.: The Nature of Statistic Learning Theory. Springer, New York (1995)CrossRefGoogle Scholar
  30. Voica, A.: PowerVR GX5300: the world’s smallest GPU for next-generation wearables and IoT. Imagination Blog. 2014. Accessed 5 Jan 2016
  31. Wang, X., et al.: Gesture recognition using mobile phone’s inertial sensors. In: Distributed Computing and Artificial Intelligence, pp. 173–184. Springer, Berlin (2012)Google Scholar
  32. Wobbrock, J.O., Wilson, A.D., Li, Y.: Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, pp. 159–168. ACM (2007)Google Scholar
  33. Wu, J., et al.: Gesture recognition with a 3-D accelerometer. In: UIC ‘09 Proceedings of the 6th International Conference on Ubiquitous Intelligence and Computing, pp. 25–38 Brisbane (2009). Accessed July 7–9 2009Google Scholar
  34. Zhang, X., et al.: A framework for hand gesture recognition based on accelerometer and EMG sensors. In: IEEE Trans. Syst. Man Cybern. Part A: Syst. Hum. 41(6), 1064–1076 (2011) Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Department of Electrical and Computer EngineeringAmerican University of BeirutBeirutLebanon

Personalised recommendations