Intelligent Intent-Aware Touchscreen Systems Using Gesture Tracking with Endpoint Prediction

  • Bashar I. AhmadEmail author
  • Patrick M. Langdon
  • Robert Hardy
  • Simon J. Godsill
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9176)


Using an interactive display, such as a touchscreen, entails undertaking a pointing gesture and dedicating a considerable amount of attention to execute a selection task. In this paper, we give an overview of the concept of intent-aware interactive displays that can determine, early in the free hand pointing gesture, the icon/item the user intends to select on the touchscreen. This can notably reduce the pointing time, aid implementing effective selection facilitation routines and enhance the overall system accuracy as well as the user experience. Intent-aware displays employ a gesture tracking sensor in conjunction with novel probabilistic intent inference algorithms to predict the endpoint of a free hand pointing gesture. Real 3D pointing data is used to illustrate the usefulness and effectiveness of the proposed approach.


Interactive displays Finger tracking Bayesian inference Target assistance Endpoint prediction 


  1. 1.
    Wu, F.G., Lin, H., You, M.: Direct-touch vs. mouse input for navigation modes of the web map. Displays 32(5), 261–267 (2011)CrossRefGoogle Scholar
  2. 2.
    Burnett, G.E., Mark Porter, J.: Ubiquitous computing within cars: designing controls for non-visual use. Int. J. Hum Comput Stud. 55(4), 521–531 (2001)CrossRefGoogle Scholar
  3. 3.
    Harvey, C., Stanton, N.A.: Usability evaluation for in-vehicle systems. CRC Press, London (2013)CrossRefGoogle Scholar
  4. 4.
    Volvo Cars, Volvo car group unveils concept estate at Geneva motor show 27th February 2014. Accessed on: 14 October 2014 from
  5. 5.
    Goel, M., Findlater, L., Wobbrock, J.: Walktype: using accelerometer data to accomodate situational impairments in mobile touch screen text entry. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2687–2696. ACM (2012)Google Scholar
  6. 6.
    Jaeger, M.G., Skov, M.B., Thomassen, N.G., et al.: You can touch, but you can’t look: interacting with in-vehicle systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1139–1148 (2008)Google Scholar
  7. 7.
    Ahmad, B.I., Langdon, P.M., Godsill, S.J., Hardy, R., Dias, E., Skrypchuk, L.: Interactive displays in vehicles: Improving usability with a pointing gesture tracker and Bayesian intent predictors. In: Proc. of International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI 2014), pp. 1–8. ACM (2014)Google Scholar
  8. 8.
    Ahmad, B.I., Murphy, J., Langdon, P.M., Godsill, S.J., Hardy, R., Skrypchuk, L.: Intent Inference for Pointing Gesture Based Interactions in Vehicles. IEEE Transactions on Cybernetics (2015)Google Scholar
  9. 9.
    Ahmad, B.I., Murphy, J.K., Langdon, P.M., Godsill, S.J.: Bayesian target prediction from partial finger tracks: Aiding interactive displays in vehicles. In: Proceedings of the 17th International Conference on Information Fusion (FUSION 2014), pp. 1–7 (2014)Google Scholar
  10. 10.
    Ahmad, B.I., Murphy, J.K., Langdon, P.M., Godsill, S.J.: Filtering perturbed in-vehicle pointing gesture trajectories: Improving the reliability of intent inference. In: Proceedings of IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2014) (2014)Google Scholar
  11. 11.
    Ahmad, B.I., Murphy, J., Langdon, P.M., Godsill, S.J.: Destination Inference Using Bridging Distributions. In: Proceedings of the 40th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2015) (2015)Google Scholar
  12. 12.
    Garber, L.: Gestural technology: moving interfaces in a new direction [technology news]. Comput. IEEE 46(10), 22–25 (2013)CrossRefGoogle Scholar
  13. 13.
    Leap Motion Website.
  14. 14.
    McGuffin, M.J., Balakrishnan, R.: Fitts’ law and expanding targets: Experimental studies and designs for user interfaces. ACM Trans. Comput.-Hum. Interact. 12(4), 388–422 (2005)CrossRefGoogle Scholar
  15. 15.
    Murata, A.: Improvement of pointing time by predicting targets in pointing with a PC mouse. IJHCI 10(1), 23–32 (1998)Google Scholar
  16. 16.
    Lane, D., Peres, S., Sandor, A., Napier, H.: A process for anticipating and executing icon selection in graphical user interfaces. IJHCI 19(2), 241–252 (2005)Google Scholar
  17. 17.
    Wobbrock, J.O., Fogarty, J., Liu, S., Kimuro, S., Harada, S.: The angle mouse: target-agnostic dynamic gain adjustment based on angular deviation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1401–1410 (2009)Google Scholar
  18. 18.
    Ahmad, B.I., Langdon, P.M., Bunch, P., Godsill, S.J.: Probabilistic intentionality prediction for target selection based on partial cursor tracks. In: Stephanidis, C., Antona, M. (eds.) UAHCI 2014, Part III. LNCS, vol. 8515, pp. 427–438. Springer, Heidelberg (2014)Google Scholar
  19. 19.
    Asano, T., Sharlin, E., Kitamura, Y., Takashima, K., Kishino, F.: Predictive interaction using the Delphian desktop. In: Proceedings of the ACM Symposium on User Interface Software and Technology, pp. 133–141. ACM (2005)Google Scholar
  20. 20.
    Lank, E., Cheng, Y.-C.N., Ruiz, J.: Endpoint prediction using motion kinematics. In: Proceedings of the SIGCHI Conf. on Human factors in Computing Systems, pp. 637–646 (2007)Google Scholar
  21. 21.
    Ziebart, B., Dey, A., Bagnell, J.A.: Probabilistic pointing target prediction via inverse optimal control. In: Proceedings of the 2012 ACM International. Conference on Intelligent User Interfaces, pp. 1–10 (2012)Google Scholar
  22. 22.
    Godsill S., Vermaak, J.: Models and algorithms for tracking using trans-dimensional sequential Monte Carlo. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2004), vol. 3, pp. 973–976. IEEE (2004)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Bashar I. Ahmad
    • 1
    Email author
  • Patrick M. Langdon
    • 1
  • Robert Hardy
    • 2
  • Simon J. Godsill
    • 1
  1. 1.Department of EngineeringUniversity of CambridgeCambridgeUK
  2. 2.Jaguar Land Rover, WhitleyCoventryUK

Personalised recommendations