Skip to main content
SpringerLink
Log in

Human interaction with IoT-based smart environments

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper describes concepts, design, implementation, and performance evaluation of a 3D-based user interface for accessing IoT-based Smart Environments (IoT-SE). The generic interaction model of the described work addresses some major challenges of Human-IoT-SE-Interaction such as cognitive overload associated with manual device selection in complex IoT-SE, loss of user control, missing system image or over-automation. To address these challenges we propose a 3D-based mobile interface for mixed-initiative interaction in IoT-SE. The 3D visualization and 3D UI, acting as the central feature of the system, create a logical link between physical devices and their virtual representation on the end user’s mobile devices. By so doing, the user can easily identify a device within the environment based on its position, orientation, and form, and access the identified devices through the 3D interface for direct manipulation within the scene. This overcomes the problem of manual device selection. In addition, the 3D visualization provides a system image for the IoT-SE, which supports users in understanding the ambience and things going on in it. Furthermore, the mobile interface allows users to control the amount and the way the IoT-SE automates the environment. For example, users can stop or postpone system triggered automatic actions, if they don’t like or want them. Users also can remove a rule forever. By so doing, users can delete smart behaviors of their IoT-SE. This helps to overcome the automation challenges. In this paper, we present the design, implementation and evaluation of the proposed interaction system. We chose smart meeting rooms as the context for prototyping and evaluating our interaction concepts. However, the presented concepts and methods are generic and could be adapted to similar environments such as smart homes. We conducted a subjective usability evaluation (ISO-Norm 9241/110) with 16 users. All in one the study results indicate that the proposed 3D-User Interface achieved a good high score according to the ISO-Norm scores.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. (2015) Allnet homepage. http://www.allnet.de/en/

  2. (2015) Ant+ homepage. http://www.thisisant.com/

  3. (2015) Digital living network alliance. http://www.dlna.org

  4. (2015) FRITZ!Box, AVM International. http://en.avm.de/products/fritzbox/

  5. (2015) Knx association. www.knx.org/

  6. (2015) Plugwise homepage. https://www.plugwise.com/

  7. (2015) Raspberry pi homepage. https://www.raspberrypi.org/

  8. (2015) Unity3d homepage. http://www.unity3d.com

  9. (2015) Upnp forum home page and standards specs. http://upnp.org/

  10. (2015) Zwave homepage. http://www.zwave.com

  11. (1996) “IEEE Draft Standard P802.11”. Wireless LAN

  12. (2004) “IEEE 802.3ah”. Ethernet in the First Mile

  13. Aart EH, Encarnação JL (2006) True Visions: The Emergence of Ambient Intelligence. Springer

  14. Aarts E, Wichert R (2009) Ambient intelligence. Springer

  15. Adrian RAA, Shirehjini N (2008) Realization of a upnp agent for controlling adaptive multimedia environments. Master’s thesis, Technische Universität Darmstadt

  16. Allouch SB, van Dijk JA, Peters O (2009) The acceptance of domestic ambient intelligence appliances by prospective users. In: Pervasive Computing. Springer, pp 77–94

  17. Amendola S, Lodato R, Manzari S, Occhiuzzi C, Marrocco G (2014) Rfid technology for iot-based personal healthcare in smart spaces. IEEE Internet of Things Journal 1(2):144–152

    Article  Google Scholar 

  18. Balta-Ozkan N, Davidson R, Bicket M, Whitmarsh L (2013) Social barriers to the adoption of smart homes. Energy Policy 63:363–374

    Article  Google Scholar 

  19. Bandyopadhyay D, Sen J (2011) Internet of things: Applications and challenges in technology and standardization. Wirel Pers Commun 58(1):49–69

    Article  Google Scholar 

  20. Belimpasakis P, Stirbu V (2014) A survey of techniques for remote access to home networks and resources. Multimedia Tools and Applications 70(3):1899–1939

    Article  Google Scholar 

  21. Bilgin A, Hagras H, Malibari A, Alghazzawi D, Alhaddad MJ (2013) A computing with words framework for ambient intelligence. In: 2013 IEEE International Conference on Systems, man, and cybernetics (SMC). IEEE, pp 2887–2892

  22. Bonino D, Corno F (2011) What would you ask to your home if it were intelligent? exploring user expectations about next-generation homes. Journal of Ambient Intelligence and Smart Environments 3(Number):111–126

    Google Scholar 

  23. Burmeister D, Altakrouri B, Schrader A (2015) Ambient reflection: Towards self-explaining devices. Large-scale and Model-based Interactive Systems:16

  24. Chen YH, Zhang B, Tuna C, Li Y, Lee EA, Hartmann B (2013) A context menu for the real world: Controlling physical appliances through head-worn infrared targeting. Tech. rep., DTIC Document

  25. Condado PA, Lobo FG (2015) A system for controlling assisted living environments using mobile devices. In: Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility. ACM, pp 33–38

  26. Gilman E, Davidyuk O, Su X, Riekki J (2013) Towards interactive smart spaces. Journal of Ambient Intelligence and Smart Environments 5(1):5–22

    Google Scholar 

  27. Gubbi J, Buyya R, Marusic S, Palaniswami M (2013) Internet of things (iot): A vision, architectural elements, and future directions. Futur Gener Comput Syst 29(7):1645–1660

    Article  Google Scholar 

  28. Hervás R, Bravo J, Fontecha J, Villarreal V (2013) Achieving adaptive augmented reality through ontological context-awareness applied to aal scenarios. Journal of Universal Computer Science 19(9):1334–1349

    Google Scholar 

  29. Hoff KA, Bashir M (2014) Trust in automation integrating empirical evidence on factors that influence trust. Human Factors: The Journal of the Human Factors and Ergonomics Society:0018720814547570

  30. Horvitz EJ (2007) Reflections on challenges and promises of mixed-initiative interaction. AI Mag 28(2):3

    Google Scholar 

  31. Hossain MA, Shirehjini AAN, Alghamdi AS, El Saddik A (2013) Adaptive interaction support in ambient-aware environments based on quality of context information. Multimedia Tools and Applications 67(2):409–432

    Article  Google Scholar 

  32. Kashyap H, Singh V, Chauhan V, Siddhi P (2015) A methodology to overcome challenges and risks associated with ambient intelligent systems. In: 2015 International Conference on Advances in Computer engineering and applications (ICACEA). IEEE, pp 245–248

  33. Loeffler D, Hess A, Maier A, Hurtienne J, Schmitt H (2013) Developing intuitive user interfaces by integrating users’ mental models into requirements engineering. In: Proceedings of the 27th International BCS Human Computer Interaction Conference. British Computer Society, p 15

  34. Martin ZAA, Shirehjini N (2008) Realization of a upnp agent for controlling adaptive multimedia environments. Master’s thesis, Technische Universität Darmstadt

  35. Mattern F (2003) Vom verschwinden des computers die vision des ubiquitous computing. In: Total Vernetzt. Springer, pp 1–41

  36. Mostafazadeh A, Shirehjini AAN, Daraei S (2015) A meta user interface for understandable and predictable interaction in aal. In: Human aspects of IT for the aged population. Design for Everyday Life. Springer, pp 456–464

  37. Nazari Shirehjini A (2007) A multidimensional classification model for the interaction in reactive media rooms. Human-Computer Interaction HCI Intelligent Multimodal Interaction Environments :431–439

  38. Oduor E, Neustaedter C (2014) The family room: A multi-camera, multi-display family media space. In: Proceedings of the Companion Publication of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, pp 289–292

  39. Petzold M, Barbabella F, Bobeth J, Kern D, Mayer C, Morandell M (2013) Towards an ambient assisted living user interaction taxonomy. In: CHI’13 Extended Abstracts on Human Factors in Computing Systems. ACM, pp 49–54

  40. Portet F, Vacher M, Golanski C, Roux C, Meillon B (2013) Design and evaluation of a smart home voice interface for the elderly: Acceptability and objection aspects. Pers Ubiquit Comput 17(1):127–144

    Article  Google Scholar 

  41. Poslad S (2011) Ubiquitous computing: smart devices, environments and interactions. Wiley

  42. Rahimi H, Shirehjini AAN, Shirmohammadi S (2011) Context-aware prioritized game streaming

  43. Sadri F (2011) Ambient intelligence: A survey. ACM Computing Surveys (CSUR) 43(4):36

    Article  Google Scholar 

  44. Savazzi S, Rampa V, Spagnolini U (2014) Wireless cloud networks for the factory of things: Connectivity modeling and layout design. IEEE Internet of Things Journal 1(2):180–195

    Article  Google Scholar 

  45. Schmitt F, Cassens J, Kindsmüller MC, Herczeg M (2011) Mental models of ambient systems: A modular research framework. In: Modeling and Using Context. Springer, pp 278–291

  46. Shirehjini AAN (2004) A novel interaction metaphor for personal environment control: Direct manipulation of physical environment based on 3d visualization. Comput Geotech 28(5):667–675

    Google Scholar 

  47. Shirehjini AAN (2005) A generic upnp architecture for ambient intelligence meeting rooms and a control point allowing for integrated 2d and 3d interaction. In: Proceedings of the 2005 joint conference on smart objects and ambient intelligence: Innovative context-aware services: Usages and Technologies. ACM, pp 207–212

  48. Shirehjini AAN (2008) Interaktion in Ambient Intelligence: Konzeption eines intuitiven assistenten zur ganzheitlichen und konfliktfreien Interaktion in adaptiven Umgebungen. PhD thesis, Technische Universität Darmstadt

  49. Shirehjini AAN, Shirmohammadi S (2009) A mobile 3d user interface for interaction with ambient audio visual environments. In: IEEE International Workshop on Haptic Audio visual Environments and games, 2009. HAVE 2009. IEEE, pp 186–191

  50. Shirehjini AAN, Yassine A, Shirmohammadi S (2012) An rfid-based position and orientation measurement system for mobile objects in intelligent environments. IEEE Trans Instrum Meas 61(6):1664–1675

    Article  Google Scholar 

  51. Shirehjini AAN, Chegini M, Shirmohammadi S (2015) The clutch: Two-handed mobile multi-touch 3d object translation and manipulation

  52. Shneiderman B (1997) Direct manipulation for comprehensible, predictable and controllable user interfaces. In: Proceedings of the 2nd International Conference on Intelligent User Interfaces. ACM, pp 33–39

  53. Shirehjini AAN, Soltaninejad F, Saniee G, Semsar A (2016) Mental model development support using collaborative 3d virtual environments. In: HCI international 2016. Toronto, ON, Canada

  54. Stankovic J, et al. (2014) Research directions for the internet of things. IEEE Internet of Things Journal 1(1):3–9

    Article  Google Scholar 

  55. Tang J, Kim S (2015) A service-oriented device selection solution based on user satisfaction and device performance in a ubiquitous environment. Multimedia Tools and Applications 74(23):10,761–10,783

    Article  Google Scholar 

  56. Vacher M, Lecouteux B, Portet F (2015) On distant speech recognition for home automation. In: Smart Health. Springer, pp 161–188

  57. Yu J, Kim M, Bang HC, Bae SH, Kim SJ (2015) Iot as a applications: Cloud-based building management systems for the internet of things. Multimedia Tools and Applications:1–14

  58. Zeidler C, Lutteroth C, Weber G (2013) An evaluation of stacking and tiling features within the traditional desktop metaphor. In: Human-Computer Interaction–INTERACT 2013. Springer , pp 702–719

Download references

Author information

Authors and Affiliations

  1. Computer Engineering Department, Sharif University of Technology, Tehran, Iran

    Ali Asghar Nazari Shirehjini & Azin Semsar

Authors
  1. Ali Asghar Nazari Shirehjini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Azin Semsar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ali Asghar Nazari Shirehjini.

Appendices

Appendix A: Message format of the AmIS protocol

The whole payload is framed by an identifier tag that represents the type of the message. In this tag the elements mid, uid, key, value and situation provide all necessary information that is transported. The mid field holds the unique message ID which is generated when the message is created. Each message can be referenced by this ID field. The next entry uid holds a unique identifier of an AmIS component, a device or a media file. This field is used to identify an entity that is the target of a command or a subscription message. Afterwards, the key entry provides the type of notification, the corresponding abstract action or the type of transaction, depending on the used message type. As last element the situation field is added. This entry is modeled as a universal field for extra information like a timestamp or anything of interest that might be added in the future.

figure c

Appendix B: Example of a projector for effect on a connected device property

figure d

Appendix C: Example of an XML file describing a room

figure e

Appendix D: Example of an XML file describing a device

For each device there is one service tag defined that describes the device model. Furthermore, capabilityentries contain the supported actions for this device, whereas the tag propertyentries describe the specific state for the device. The URL-base is a URL to an archive file that contains the 3D Model and textures as well as the Device Control Interfaces for the described device. The capabilityentries tag describes those attributes of a device that are changed when invoking a specific action. For example the action Dimmer DIMM affects the attribute lightlevel of the current device.

figure f

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nazari Shirehjini, A.A., Semsar, A. Human interaction with IoT-based smart environments. Multimed Tools Appl 76, 13343–13365 (2017). https://doi.org/10.1007/s11042-016-3697-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3697-3

Keywords

Search

Navigation