Skip to main content
SpringerLink
Log in

“Point at It with Your Smartphone”: Assessing the Applicability of Orientation Sensing of Smartphones to Operate IoT Devices

  • Conference paper
  • First Online:
HCI International 2021 - Late Breaking Papers: Multimodality, eXtended Reality, and Artificial Intelligence (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 13095))

Included in the following conference series:

Abstract

The built-in orientation and motion sensors of smartphones along with their wireless communication abilities are utilized to control connected IoT devices from any place in a room, by pointing at them with the smartphone in the hand. The information of which device is targeted will be derived from the user’s actual location, the spatial orientation of the smartphone and pre-knowledge regarding the positions of devices. Chosen devices are remotely operated with simple mid-air gestures performed with the smartphone. The feasibility of this cost-effective approach is assessed by user experiments. The continuous readings of the smartphone’s inclination, rotation and magnetic field sensors are recorded with a dedicated freeware app. An algorithm combines the sensor readings to deliver the actual spatial orientation. Our preliminary experiments with different smartphone models and several users show that pointing at defined positions and performing gestures with a smartphone in the user’s hand can be accurately sensed without latency and with small deviations of the orientation measurements in the range of up to 5 degrees, indicating the feasibility of this novel approach.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Voice user interfaces, while easier to use, can show annoying performance drops due to disturbances from ambient noise or unclear pronunciation. Moreover, they meet reservations from the hesitation to speak to a technical device altogether and raise concerns regarding data privacy issues.

  2. 2.

    Physics Toolbox Sensor Suite, https://www.vieyrasoftware.net/.

References

  1. Clark, H.H.: Pointing and placing. In: Kita, S. (ed.) Pointing: Where Language, Culture, and Cognition Meet, pp. 243–268. Taylor & Francis (2003)

    Google Scholar 

  2. Cartmill, E.A., Beilock, S., Goldin-Meadow, S.: A word in the hand: action, gesture and mental representation in humans and non-human primates. Philos. Trans. R. Soc Part B 367(1585), 129–143 (2012)

    Article  Google Scholar 

  3. Wong, N., Gutwin, C.: Where are you pointing? The accuracy of deictic pointing in CVEs. In: Proceedings of the CHI, pp. 1029–1038 (2010)

    Google Scholar 

  4. Cockburn, A., Quinn, P., Gutwin, C., Ramos, G., Looser, J.: Air pointing: design and evaluation of spatial target acquisition with and without visual feedback. Int. J. Hum. Comput. Stud., 401–414 (2011)

    Google Scholar 

  5. Akkil, D., Isokoski, P.: Accuracy of interpreting pointing gestures in egocentric view. In: Ubicomp 2016, Heidelberg (2016)

    Google Scholar 

  6. Flanagin, V.L., Fisher, P., Olcay, B., Kohlbecher, S., Brandt, T.: A bedside application-based assessment of spatial orientation and memory: approaches and lessons learned. J. Neurol. 266(1), 126–138 (2019). https://doi.org/10.1007/s00415-019-09409-7

    Article  Google Scholar 

  7. Kirsh, I., Ruser, H.: Phone-pointing remote app: using smartphones as pointers in gesture-based IoT remote controls. In: Stephanidis, C., Antona, M., Ntoa, S. (eds.) HCII 2021. CCIS, vol. 1420, pp. 14–21. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-78642-7_3

  8. Jota, R., Nacenta, M. Jorge, J., Carpendale, Sh., Greenberg, S: A comparison of ray pointing techniques for very large displays. In: ACM 36th Graphics Interface Conference, Ottawa (2010)

    Google Scholar 

  9. Nancel, M., Chapuis, O., Pietriga, E., Yang, X., Irani, P., Beaudouin-Lafon, M.: High-precision pointing on large wall displays using small handheld devices. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI 2013), pp. 831–840 (2013)

    Google Scholar 

  10. Wilson, A., Shafer St.: XWand: UI for intelligent spaces. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI 2003), pp. 545–552 (2003)

    Google Scholar 

  11. Ouchi, K., Esaka, N., Tamura, Y., Hirahara, M., Doi, M.: MagicWand: an intuitive gesture remote control for home appliances. In: IEEE International Conference on Active Media Technology (2005)

    Google Scholar 

  12. Spindler, M., Schuessler, M., Martsch, M., Dachselt R: Move your phone: spatial input-based document zoom & pan on mobile displays revisited. In: CHI 2014 Extended Abstracts on Human Factors in Computing Systems (CHI EA 2014), pp. 515–518 (2014)

    Google Scholar 

  13. Seifert, J., Bayer, A., Rukzio, E.: PointerPhone: using mobile phones for direct pointing interactions with remote displays. In: Kotzé, P., Marsden, G., Lindgaard, G., Wesson, J., Winckler, M. (eds.) INTERACT 2013. LNCS, vol. 8119, pp. 18–35. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40477-1_2

  14. Pietroszek, K., Kuzminykh, A., Wallace, J.R., Lank E.: Smartcasting: a discount 3D interaction technique for public displays. In: 26th Australian Computer-Human Interaction Conference on Designing Futures (OzCHI 2014), pp. 119–128 (2014)

    Google Scholar 

  15. Pietroszek, K., Wallace, J., Lank, E.: Tiltcasting: 3D interaction on large displays using a mobile device. In: Proceedings of the 28th ACM Symposium on User Interface Software & Technology (UIST 2015), pp. 57–62 (2015)

    Google Scholar 

  16. Hartmann, J., Vogel, D.: An evaluation of mobile phone pointing in spatial augmented reality. In CHI 2018 Extended Abstracts on Human Factors in Computing Systems (CHI EA 2018) (2018)

    Google Scholar 

  17. Plaumann, K., Weing, M., Winkler, C., Müller, M., Rukzio, E.: Towards accurate cursorless pointing: the effects of ocular dominance and handedness. Pers. Ubiquit. Comput. 22(4), 633–646 (2017). https://doi.org/10.1007/s00779-017-1100-7

    Article  Google Scholar 

  18. Hosseinianfar, H., Chizari, A., Salehi, J.A.: GOPA: geometrical optics positioning algorithm using spatial color coded LEDs. arXiv:1807.06931v1 (2018)

  19. Sato, K., Matsushita, M.: Object manipulation by absolute pointing with a smartphone gyro sensor. In: ACM Symposium on Spatial User Interaction (SUI 2019), New Orleans (2019)

    Google Scholar 

  20. Alce, G., Espinoza, A., Hartzell, T., Olsson, St., Samuelsson, D., Wallergård, M.: UbiCompass: an IoT interaction concept. Adv. Hum. Comput. Interact. 2018, Article ID 5781363 (2018)

    Google Scholar 

  21. Solin, A., Cortes, S., Rahtu, E., Kannala, J. Inertial odometry on handheld smartphones. In: 21st IEEE International Conference on Information Fusion (FUSION), pp. 1–5 (2018)

    Google Scholar 

  22. Gromov, B., Abbate, G., Gambardella, L., Giusti A.: Proximity human-robot interaction using pointing gestures and a wrist-mounted IMU. In: International Conference on Robotics and Automation (ICRA) (2019)

    Google Scholar 

  23. Odenwald, S.: Smartphone sensors for citizen science applications: radioactivity and magnetism. Citiz. Sci. Theory Pract. 4(1), 18, pp. 1–15 (2019)

    Google Scholar 

  24. Umek, A., Kos, A.: Validation of smartphone gyroscopes for mobile biofeedback applications. Pers. Ubiquitous Comput. 20, 657–666 (2016). https://doi.org/10.1007/s00779-016-0946-4

  25. Correa, A., Barcelo, M., Morell, A., Vicario, J.L.: A review of pedestrian indoor positioning systems for mass market applications. Sensors 17(8), ID 1927 (2017)

    Google Scholar 

  26. Kuhlmann, T., Garaizar, P., Reips, U.-D.: Smartphone sensor accuracy varies from device to device in mobile research: the case of spatial orientation. Behav. Res. Methods 53(1), 22–33 (2020). https://doi.org/10.3758/s13428-020-01404-5

    Article  Google Scholar 

  27. Kela, J., Korpipää, P., Mäntyjärvi, J., Kallio, S., Savino, G.: Accelerometer-based gesture control for a design environment. ACM Pers. Ubiquitous Comput. 10(5), 285–299 (2006)

    Article  Google Scholar 

  28. Agrawal, S., Constandache, I., Gaonkar, Sh., Choudhury, R.R.: Using mobile phones to write in air. In: International Conference on Mobile Systems, Applications, and Services (MobiSys 2011), pp. 15–28. ACM (2011)

    Google Scholar 

  29. Kok, M., Schön, T.: Magnetometer calibration using inertial sensors. IEEE Sens. J. 16(14), 5679–5689 (2016)

    Article  Google Scholar 

  30. Li, X., Li, Z.: A new calibration method for tri-axial field sensors in strap-down navigation systems. Meas. Sci. Technol. 23(10), 105105 (2012)

    Google Scholar 

  31. Salehi, S., Mostofi, N., Bleser, G.: A practical in-field magnetometer calibration method for IMUs. In: Proceedings of the IROS Workshop on Cognitive Assistive Systems: Closing the Action-Perception Loop, pp. 39–44 (2012)

    Google Scholar 

  32. Langlois, Ch., Tiku, S, Pasricha, S.: Indoor localization with smartphones. IEEE Consum. Electron. Mag. 10(17), 70–80 (2017)

    Google Scholar 

  33. Nguyen, K.A., Luo, Zh., Li, G., Watkins, Ch.: A review of smartphones based indoor positioning: challenges and applications. IET Cybersyst. Robotics 3(1), 1–30 (2021)

    Google Scholar 

  34. Lymberopoulos, D., Liu, J.: The microsoft indoor localization competition: experiences and lessons learned. IEEE Signal Process. Mag. 09(17), 125–140 (2017)

    Article  Google Scholar 

  35. Ashraf, I., Hur, S., Park, Y.: Indoor positioning on disparate commercial smartphones using wi-fi access points coverage area. Sensors 19(19), 4351 (2019)

    Article  Google Scholar 

  36. Retscher, G.: Fundamental concepts and evolution of wi-fi user localization: an overview based on different case studies. Sensors 20(18), 5121 (2020)

    Article  Google Scholar 

  37. Khalajmehrabadi, A., Gatsis, N., Akopian, D.: Modern WLAN fingerprinting indoor positioning methods and deployment challenges. IEEE Commun. Surv. Tutor. 19(3), 1974–2002 (2017)

    Article  Google Scholar 

  38. Real Ehrlich, C., Blankenbach J.: Indoor localization for pedestrians with real-time capability using multi-sensor smartphones. Geo-spat. Inf. Sci. (2019). https://doi.org/10.1080/10095020.2019.1613778

  39. Pascacio, P., Casteleyn, S., Torres-Sospedra, J.: Collaborative indoor positioning systems: a systematic review. Sensors 21(3), 1002 (2021)

    Article  Google Scholar 

  40. Landau, Y., Ben-Moshe, B.: STEPS: an indoor navigation framework for mobile devices. Sensors 20(14), 3929 (2020)

    Article  Google Scholar 

  41. Mezari, A., Maglogiannis, I.; An easily customized gesture recognizer for assisted living using commodity mobile devices. J. Healthc. Eng. 2018, Article ID 3180652 (2018)

    Google Scholar 

  42. Madapana, N., Gonzalez, G., Zhang, L., Rodgers, R., Wachs, J.: Agreement study using gesture description analysis. IEEE Trans. Hum. Mach. Syst. 50(5), 434–443 (2020)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Bundeswehr University Munich, Neubiberg, Germany

    Heinrich Ruser

  2. The Academic College of Tel Aviv-Yafo, Tel Aviv, Israel

    Ilan Kirsh

Authors
  1. Heinrich Ruser
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ilan Kirsh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Heinrich Ruser .

Editor information

Editors and Affiliations

  1. University of Creteand Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

  2. The Open University of Japan, Chiba, Japan

    Masaaki Kurosu

  3. U.S. Army Research Laboratory, Aberdeen Proving Ground, MD, USA

    Jessie Y. C. Chen

  4. U.S. Army Combat Capabilities Development Command Soldier Center, Orlando, FL, USA

    Gino Fragomeni

  5. Smart Future Initiative, Frankfurt am Main, Hessen, Germany

    Norbert Streitz

  6. Faculty of Arts and Science, Kyushu University, Fukuoka, Japan

    Shin'ichi Konomi

  7. Siemens (United States), Princeton, NJ, USA

    Helmut Degen

  8. Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Stavroula Ntoa

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ruser, H., Kirsh, I. (2021). “Point at It with Your Smartphone”: Assessing the Applicability of Orientation Sensing of Smartphones to Operate IoT Devices. In: Stephanidis, C., et al. HCI International 2021 - Late Breaking Papers: Multimodality, eXtended Reality, and Artificial Intelligence. HCII 2021. Lecture Notes in Computer Science(), vol 13095. Springer, Cham. https://doi.org/10.1007/978-3-030-90963-5_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-90963-5_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-90962-8

  • Online ISBN: 978-3-030-90963-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation