Abstract
Remote control mobile applications for operating Internet of Things (IoT) devices using smartphones are commonly based on a touch user interface. The effort of using such apps is often disproportionate to the simplicity of carrying out the actions manually. For example, turning a light on or off via menus and forms of a standard remote app might not be very convenient. A voice user interface, while easier to use, gives rise to other issues, including user privacy and distracting others nearby. This paper proposes a new type of universal IoT remote control applications for smartphones: phone-pointing remote apps. Using a phone-pointing remote app, users can physically point their smartphones at IoT devices to select them, and operate them via movement gestures, without needing to turn on the phone screen or talk, and with no need for any additional hardware. This new approach provides a unique combination of advantages. It is simple, intuitive, fast, and voiceless. Instead of using the touchscreen or the microphone as the input source, phone-pointing remote apps will use a combination of standard smartphone sensors, including the GNSS sensor, Wi-Fi scanner, Bluetooth receiver, camera, barometer, magnetometer, g-force meter, accelerometer, and gyroscope. An analysis of the proposed model in light of relevant results from related studies provides positive preliminary indications regarding the feasibility of this novel approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Acebron, J., Spigler, R.: The remote control and beyond: the legacy of Robert Adler. SIAM News 40(5), 1–2 (2007)
Bai, Y., Su, H., Hsu, W.: Indoor and remote controls and management of home appliances by a smartphone with a four-quadrant user interface. In: IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, USA, pp. 319–320 (2017). https://doi.org/10.1109/ICCE.2017.7889336
Hoy, M.B.: Alexa, Siri, Cortana, and more: an introduction to voice assistants. Med. Ref. Serv. Q. 37(1), 81–88 (2018). https://doi.org/10.1080/02763869.2018.1404391
Khan, A., Al-Zahrani, A., Al-Harbi, S., Al-Nashri, S., Khan, I.A.: Design of an IoT smart home system. In: 15th Learning and Technology Conference (L&T), pp. 1–5 (2018). https://doi.org/10.1109/LT.2018.8368484
Kuhlmann, T., Garaizar, P., Reips, U.D.: Smartphone sensor accuracy varies from device to device in mobile research: the case of spatial orientation. Behav. Res. Methods 53(1), 22–33 (2020). https://doi.org/10.3758/s13428-020-01404-5
Murata, M., Ahmetovic, D., Sato, D., Takagi, H., Kitani, K.M., Asakawa, C.: Smartphone-based indoor localization for blind navigation across building complexes. In: 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom), pp. 1–10. IEEE (2018)
Ruser, H., Kirsh, I.: Point at it with your smartphone: assessing the applicability of orientation sensing of smartphones to operate IoT devices. In: Proceedings of the 23rd HCI International Conference (HCII). Springer, Cham (2021)
Ruser, H., Vorwerg, S., Eicher, C.: Making the home accessible - experiments with an infrared handheld gesture-based remote control. In: Stephanidis, C., Antona, M. (eds.) HCII 2020. CCIS, vol. 1226, pp. 89–97. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50732-9_13
Sayago, S., Neves, B.B., Cowan, B.R.: Voice assistants and older people: some open issues. In: Proceedings of the 1st International Conference on Conversational User Interfaces (CUI), pp. 1–3, New York, USA. Association for Computing Machinery (2019). https://doi.org/10.1145/3342775.3342803
Shin, B.J., Lee, K.W., Choi, S.H., Kim, J.Y., Lee, W.J., Kim, H.S.: Indoor wifi positioning system for android-based smartphone. In: International Conference on Information and Communication Technology Convergence (ICTC), pp. 319–320 (2010). https://doi.org/10.1109/ICTC.2010.5674691
Terzopoulos, G., Satratzemi, M.: Voice assistants and smart speakers in everyday life and in education. Inf. Educ. 19, 473–490 (2020). https://doi.org/10.15388/infedu.2020.21
Wang, X., Tarrío, P., Metola, E., Bernardos, A.M., Casar, J.R.: Gesture recognition using mobile phone’s inertial sensors. In: Omatu, S., De Paz Santana, J.F., González, S.R., Molina, J.M., Bernardos, A.M., Rodríguez, J.M.C. (eds.) Distributed Computing and Artificial Intelligence. AISC, vol. 151, pp. 173–184. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28765-7_21
Werner, M., Kessel, M., Marouane, C.: Indoor positioning using smartphone camera. In: International Conference on Indoor Positioning and Indoor Navigation (IPIN), pp. 1–6 (2011). https://doi.org/10.1109/IPIN.2011.6071954
Xie, C., Luan, S., Wang, H., Zhang, B.: Gesture recognition benchmark based on mobile phone. In: You, Z. (ed.) CCBR 2016. LNCS, vol. 9967, pp. 432–440. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46654-5_48
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Kirsh, I., Ruser, H. (2021). Phone-Pointing Remote App: Using Smartphones as Pointers in Gesture-Based IoT Remote Controls. In: Stephanidis, C., Antona, M., Ntoa, S. (eds) HCI International 2021 - Posters. HCII 2021. Communications in Computer and Information Science, vol 1420. Springer, Cham. https://doi.org/10.1007/978-3-030-78642-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-78642-7_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78641-0
Online ISBN: 978-3-030-78642-7
eBook Packages: Computer ScienceComputer Science (R0)