iPanel: A Computer-Vision Based Solution for Interactive Keyboard and Mouse

  • H. Chathushka Dilhan Hettipathirana
  • Pragathi Weerakoon
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8511)


This paper represents an implementation of a computer vision based interface; iPanel which employs an arbitrary panel and tip pointers as a spontaneous, wireless and mobility device. Also the proposed system can accurately identify the tip movements of the panel and simulate the relevant events on the target environment. By detecting the key pressing, mouse clicking and dragging actions, the system can fulfill many tasks. Therefore, it enables users to use their fingers naturally to interact with any application as well as with any mobility enabled devices.


Computer vision Human computer interaction gesture recognition optical character recognition wearable computing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Birchfield, S.: Elliptical head tracking using intensity gradients and color histograms. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 23-25, pp. 232–237. IEEE, Stanford University (1998)Google Scholar
  2. 2.
    Dionisio, C.R.P., Cesar Jr., R.M.: A project for hand gesture recognition. In: Symposium on Computer Graphics and Image Processing, p. 345. IEEE, Sao Paulo University (2000)Google Scholar
  3. 3.
    Desimone, R., Albright, T.D., Gross, C.G., Bruce, C.: Stimulus-Selective Properties in Inferior Temporal Neurons in the Macaque. Journal of Neuroscience 4(8), 2051–2062 (1984)Google Scholar
  4. 4.
    Dix, A., Finlay, J.E., Abowd, G.D., Beale, R.: Human-Computer Interaction, 3rd edn. Peaeson, New Delhi (2003)Google Scholar
  5. 5.
    Freeman, W.T., Anderson, D.B., Beardsley, P.A., Dodge, C.N., Roth, M., Weissman, C.D., Yerazunis, W.S.: Computer Vision for Interactive Computer Graphics. In: Computer Graphics and Applications. IEEE Computer Graphics and Applications, pp. 42–53 (May-June 1998)Google Scholar
  6. 6.
    Harding, P.R.G., Ellis, T.: Recognition Hand Gesture Using Fourier Descriptors. In: IEEE Pattern Recognition, August 23-26, vol. 3, pp. 286–289. Buckinghamshire Chilterns Univ. Coll., UK (2004)Google Scholar
  7. 7.
    Harper, R., Rodden, T., Rogers, Y., Sellen, A.: Being Human - human-computer interaction in the year 2020. Microsoft Research, Cambridge (2008)Google Scholar
  8. 8.
    Ikizler, N., Duygulu, P.: Human Action Recognition Using Distribution of Oriented Rectangular Patches. In: Elgammal, A., Rosenhahn, B., Klette, R. (eds.) Human Motion 2007. LNCS, vol. 4814, pp. 271–284. Springer, Heidelberg (2007)Google Scholar
  9. 9.
    Kendon, A.: Cross-cultural perspectives in nonverbal communication. In: How Gestures can Become Like Words, pp. 131–141 (1988)Google Scholar
  10. 10.
    Kurata, T., Okuma, T., Kourogi, M., Sakaue, K.: The Hand Mouse: GMM Hand-color Classification and Mean Shift Tracking. In: Second Intl. Workshop on Recognition, Analysis and Tracking of Faces and Gestures in Real-time Systems (July 2001)Google Scholar
  11. 11.
    Microsoft Cooperation: Microsoft Surface. Microsoft (2012), (accessed October 08, 2012)
  12. 12.
    Ong, E.J., Bowden, R.: A Boosted Classifier Tree for Hand Shape Detection. In: Proc. IEEE Intl. Conference on Automatic Face and Gesture Recognition, pp. 889–894. IEEE (2004)Google Scholar
  13. 13.
    Segen, J., Kumar, S.: GestureVR: Vision-Based 3D Hand Interface for Spatial Interaction. In: The Sixth ACM Intl. Multimedia Conference. ACM (September 1998)Google Scholar
  14. 14.
    Shen, W., Wu, L.: A method of billiard objects detection based on Snooker game video. In: Future Computer and Communication (ICFCC), Beijing, China, May 21-24, vol. 2, pp. 251–255. IEEE (2010)Google Scholar
  15. 15.
    Turk, M.: Computer Vision in the Interface. ACM Communications 47(1), 60–67 (2004)CrossRefGoogle Scholar
  16. 16.
    Vafadar, M., Behrad, A.: Human Hand Gesture Recognition using image processing for Human-Computer Interaction. In: IEEE Information and Knowledge Technology (2007)Google Scholar
  17. 17.
    Vafadar, M., Behrad, A.: Human Hand Gesture Recognition Using Motion Orientation Histogram for Interaction of Handicapped Persons with Computer. In: IEEE Image and Signal Processing, pp. 378–385 (2008)Google Scholar
  18. 18.
    Wilson, A., Shafer, S.: UI for Intelligent Spaces. In: XWand. ACM (2003)Google Scholar
  19. 19.
    Wu, Y., Huang, T.S.: View-independent recognition of hand postures. In: IEEE Computer Vision and Pattern Recognition. Beckman Inst. for Adv. Sci. & Technol., vol. 2, pp. 88–94. Illinois Univ., Urbana (2000)Google Scholar
  20. 20.
    Yang, M.H., Kriegman, D.J., Ahuja, N.: Detecting Faces in Images: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(1), 34–58 (2002)CrossRefGoogle Scholar
  21. 21.
    Zhu, X., Yang, J., Waibel, A.: Segmenting Hands of Arbitrary Color. In: Proc. IEEE Intl. Conference on Automatic Face and Gesture Recognition. Interactive Syst. Labs., pp. 446–453. Carnegie Mellon Univ., Pittsburgh (2000)Google Scholar
  22. 22.
    Bretzner, L., Laptev, I., Lindeberg, T.: Hand Gesture Recognition using Multi-Scale Colour Features, Hierarchical Models and Particle Filtering. In: Proc. IEEE Intl. Conference on Automatic Face and Gesture Recognition, pp. 423–428. IEEE, Washington, D.C (2002)CrossRefGoogle Scholar
  23. 23.
    Shi, J., Malik, J.: Motion segmentation and tracking using normalized cuts. In: Sixth International Conference on Proc. Computer Vision, January 4-7, pp. 1154–1160. IEEE, Berkeley (1998)Google Scholar
  24. 24.
    Quek, F.K.H.: Unencumbered Gestural Interaction. IEEE Multimedia 4(3), 36–47 (1996)CrossRefGoogle Scholar
  25. 25.
    Stiefelhagen, R., Yang, J.: Gaze tracking for multimodal human-computer interaction. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 1997, Karlsruhe University, April 21-24, vol. 4, pp. 2617–2620. IEEE (1997)Google Scholar
  26. 26.
    Kjeldsen, R., Kender, J.: Finding Skin in Color Images. In: Proceedings of the International Conference on Automatic Face and Gesture Recognition, pp. 312–317 (October 1996)Google Scholar
  27. 27.
    Zarit, B.D., Super, B.J., Quek, F.K.H.: Comparison of Five Color Models in Skin Pixel Classification. In: Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems, Illinois University, Chicago, IL, pp. 58–63. IEEE (September 1999)Google Scholar
  28. 28.
    Jones, M.J., Rehg, J.M.: Statistical Color Models with Application to Skin Detection. Int. Journal of Computer Vision 46(1), 81–96 (2002)CrossRefzbMATHGoogle Scholar
  29. 29.
    Comaniciu, D., Ramesh, V., Meer, P.: Real-Time Tracking of Non-Rigid Objects Using Mean Shift. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 142–149. IEEE (2000)Google Scholar
  30. 30.
    Zhu, Q., Cheng, K.T., Wu, C.T., Wu, Y.L.: Adaptive Learning of an Accurate Skin-Color Model. In: Proc. IEEE Intl. Conference on Automatic Face and Gesture Recognition, Santa Barbara, CA, USA, May 17-19, pp. 37–42. IEEE (2004)Google Scholar
  31. 31.
    Hamada, Y., Shimada, N., Shirai, Y.: Hand Shape Estimation Using Sequence of Multi-Ocular Images Based on Transition Network. In: VI 2002 (2002)Google Scholar
  32. 32.
    Gdalyahu, Y., Weinshall, D.: Flexible Syntactic Matching of Curves and its Application to Automatic Hierarchical Classification of Silhouettes. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(12), 1312–1328 (1999)CrossRefGoogle Scholar
  33. 33.
    Heap, T., Hogg, D.: Towards 3D Hand Tracking Using a Deformable Model. In: Proc. IEEE Intl. Conference on Automatic Face and Gesture Recognition, Leeds University, October 14-16, pp. 140–145. IEEE (1996)Google Scholar
  34. 34.
    Athitsos, V., Sclaroff, S.: Estimating 3D Hand Pose from a Cluttered Image. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Boston University, MA, USA, June 18-20, vol. 2, pp. 432–439. IEEE (2003)Google Scholar
  35. 35.
    Thayananthan, A., Stenger, B., Torr, P.H.S., Cipolla, R.: Shape Context and Chamfer Matching in Cluttered Scenes. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Madison, USA, June 18-20, vol. 1, pp. 127–133. IEEE (2003)Google Scholar
  36. 36.
    Xie, J.: Optical Character Recognition Based on Least Square Support Vector Machine. In: Intelligent Information Technology Application, IITA 2009, School of Electronics, Jiangxi University of Finance and Economics, Nanchang, China, November 21-22, vol. 1, pp. 626–629. IEEE (2009)Google Scholar
  37. 37.
    Harrison, C., Benko, H., Wilson, A.D.: OmniTouch: Wearable Multi-touch Interaction Everyware, pp. 16–19. ACM, Santa Barbara (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • H. Chathushka Dilhan Hettipathirana
    • 1
  • Pragathi Weerakoon
    • 2
  1. 1.Department of Computing, Informatics Institute of TechnologyCollaboration with University of WestminsterSri Lanka
  2. 2.Informatics Institute of TechnologySri Lanka

Personalised recommendations