Smartphone Input Using Its Integrated Projector and Built-In Camera

  • Sergiu Dotenco
  • Timo Götzelmann
  • Florian Gallwitz
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8512)


Touch input on modern smartphones can be tedious, especially if the touchscreen is small. Smartphones with integrated projectors can be used to overcome this limitation by projecting the screen contents onto a surface, allowing the user to interact with the projection by means of simple hand gestures. In this work, we propose a novel approach for projector smartphones that allows the user to remotely interact with the smartphone screen via its projection. We detect user’s interaction using the built-in camera, and forward detected hand gestures as touch input events to the operating system. In order to avoid costly computations, we additionally use built-in motion sensors. We verify the proposed method using an implementation for the consumer smartphone Samsung Galaxy Beam equipped with a deflection mirror.


Mobile computing user interfaces small screen fat finger problem touch input fingertip detection projector smartphone DLP projector 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bradski, G.: Programmer’s toolchest – the OpenCV library. Dr Dobb’s Journal-Software Tools for the Professional Programmer 25(11), 120–126 (2000)Google Scholar
  2. 2.
    Canny, J.: A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 8(6), 679–698 (1986), CrossRefGoogle Scholar
  3. 3.
    Colle, H.A., Hiszem, K.J.: Standing at a kiosk: Effects of key size and spacing on touch screen numeric keypad performance and user preference. Ergonomics 47(13), 1406–1423 (2004)CrossRefGoogle Scholar
  4. 4.
    Douglas, D.H., Peucker, T.K.: Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartographica: The International Journal for Geographic Information and Geovisualization 10(2), 112–122 (1973)CrossRefGoogle Scholar
  5. 5.
    Fai, A.Y.S.: Projected display shared workspaces. Patent US 20110197147 A1 (2011)Google Scholar
  6. 6.
    Gartner: Gartner says worldwide sales of mobile phones declined 3 percent in third quarter of 2012; smartphone sales increased 47 percent (2012),
  7. 7.
    Götzelmann, T.: Concept of the joint use of smartphone camera and projector for keyboard inputs. In: Proceedings of the 3rd International Symposium on Computing in Science & Engineering, ISCSE 2013, pp. 52–57. Gediz University Press, Gediz (2013a)Google Scholar
  8. 8.
    Götzelmann, T.: Deflection mirror for projector smartphones (stl file for 3d printers) (2013),
  9. 9.
    Harrison, C., Benko, H., Wilson, A.D.: Omnitouch: Wearable multitouch interaction everywhere. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST 2011, pp. 441–450. ACM, New York (2011), Google Scholar
  10. 10.
    Hasan, M.M., Mishra, P.K.: Real time fingers and palm locating using dynamic circle templates. International Journal of Computer Applications 41(6), 33–43 (2012)CrossRefGoogle Scholar
  11. 11.
    Jiang, X.-H., Li, J.-W., Wang, K.-Q., Pang, Y.-W.: A robust method of fingertip detection in complex background. In: International Conference on Machine Learning and Cybernetics (ICMLC), vol. 4, pp. 1468–1473 (2012)Google Scholar
  12. 12.
    Mistry, P., Maes, P., Chang, L.: Wuw - wear ur world: a wearable gestural interface. In: CHI 2009 Extended Abstracts on Human Factors in Computing Systems, CHI EA 2009, pp. 4111–4116. ACM, New York (2009)Google Scholar
  13. 13.
    Oka, K., Sato, Y., Koike, H.: Real-time tracking of multiple fingertips and gesture recognition for augmented desk interface systems. In: Fifth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 429–434. IEEE (2002)Google Scholar
  14. 14.
    Rahman, N.A.B.A., Wei, K.C., See, J.: RGB-H-CbCr Skin Colour Model for Human Face Detection. In: Proceedings of the MMU International Symposium on Information & Communication Technologies (2006)Google Scholar
  15. 15.
    Raskar, R., Beardsley, P.: A self-correcting projector. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. II-504–II-508 (2001)Google Scholar
  16. 16.
    Winkler, C., Pfeuffer, K., Rukzio, E.: Investigating mid-air pointing interaction for projector phones. In: Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces, ITS 2012, pp. 85–94. ACM Press, New York (2012)Google Scholar
  17. 17.
    Zhang, D., Adipat, B.: Challenges, methodologies, and issues in the usability testing of mobile applications. International Journal of Human-Computer Interaction 18(3), 293–308 (2005)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Sergiu Dotenco
    • 1
  • Timo Götzelmann
    • 1
  • Florian Gallwitz
    • 1
  1. 1.Nuremberg Institute of TechnologyGermany

Personalised recommendations