Skip to main content
SpringerLink
Log in

Programming IoT Devices by Demonstration Using Mobile Apps

  • Conference paper
  • First Online:
Book cover End-User Development (IS-EUD 2017)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 10303))

Included in the following conference series:

Abstract

The revolutionary advances of Internet of Things (IoT) devices and applications have helped IoT emerge as an increasingly important domain for end-user development (EUD). Past research has shown that end users desire to create various customized automations, which would often utilize multiple IoT devices. Many solutions exist to support EUD across multiple IoT devices, but they are limited to devices from the same manufacturer, within the same “eco-system” or supporting a common API. We present Epidosite, a mobile programming-by-demonstration system that addresses this limitation by leveraging the smartphone as a hub for IoT automation. It enables the creation of automations for most consumer IoT devices on smartphones by demonstrating the desired behaviors through directly manipulating the corresponding smartphone app for each IoT device. Epidosite also supports using the smartphone app usage context and external web services as triggers and data for automations, enabling the creation of highly context-aware IoT applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://us.smartthinq.com/.

  2. 2.

    Epidosite is a type of rock. Here, the name stands for “Enabling Programming of IoT Devices On Smartphone Interfaces for The End-users”.

  3. 3.

    https://jersey.java.net/.

  4. 4.

    https://grizzly.java.net/.

  5. 5.

    http://developer.bezirk.com/.

References

  1. Argall, B.D., et al.: A survey of robot learning from demonstration. Robot. Auton. Syst. 57(5), 469–483 (2009)

    Article  Google Scholar 

  2. Barricelli, B.R., Valtolina, S.: A visual language and interactive system for end-user development of internet of things ecosystems. J. Vis. Lang. Comput. (2017, in press)

    Google Scholar 

  3. Barricelli, B.R., Valtolina, S.: Designing for end-user development in the Internet of Things. In: Díaz, P., Pipek, V., Ardito, C., Jensen, C., Aedo, I., Boden, A. (eds.) IS-EUD 2015. LNCS, vol. 9083, pp. 9–24. Springer, Cham (2015). doi:10.1007/978-3-319-18425-8_2

    Chapter  Google Scholar 

  4. Billard, A., et al.: Robot programming by demonstration. In: Siciliano, B., Khatib, O. (eds.) Springer Handbook of Robotics, pp. 1371–1394. Springer, Heidelberg (2008)

    Google Scholar 

  5. Blackstock, M., Lea, R.: IoT interoperability: a hub-based approach. In: 2014 International Conference on the Internet of Things (IOT), pp. 79–84. IEEE (2014)

    Google Scholar 

  6. Böhmer, M., et al.: What’s in the apps for context? Extending a sensor for studying app usage to informing context-awareness. In: Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, pp. 1423–1426. ACM, New York (2013)

    Google Scholar 

  7. Coutaz, J., Crowley, J.L.: A first-person experience with end-user development for smart homes. IEEE Pervasive Comput. 15(2), 26–39 (2016)

    Article  Google Scholar 

  8. Cypher, A., Halbert, D.C.: Watch What I Do: Programming by Demonstration. MIT Press, Cambridge (1993)

    Google Scholar 

  9. Danado, J., Paternò, F.: Puzzle: a visual-based environment for end user development in touch-based mobile phones. In: Winckler, M., Forbrig, P., Bernhaupt, R. (eds.) HCSE 2012. LNCS, vol. 7623, pp. 199–216. Springer, Heidelberg (2012). doi:10.1007/978-3-642-34347-6_12

    Chapter  Google Scholar 

  10. De Russis, L., Corno, F.: HomeRules: a tangible end-user programming interface for smart homes. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp. 2109–2114. ACM, New York (2015)

    Google Scholar 

  11. Demeure, A., Caffiau, S., Elias, E., Roux, C.: Building and using home automation systems: a field study. In: Díaz, P., Pipek, V., Ardito, C., Jensen, C., Aedo, I., Boden, A. (eds.) IS-EUD 2015. LNCS, vol. 9083, pp. 125–140. Springer, Cham (2015). doi:10.1007/978-3-319-18425-8_9

    Chapter  Google Scholar 

  12. Dey, A.K., et al.: A CAPpella: programming by demonstration of context-aware applications. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 33–40. ACM (2004)

    Google Scholar 

  13. Dey, A.K., et al.: iCAP: interactive prototyping of context-aware applications. In: Fishkin, K.P., et al. (eds.) Pervasive Computing, pp. 254–271. Springer, Berlin Heidelberg (2006)

    Chapter  Google Scholar 

  14. Edwards, W.K., Grinter, R.E.: At home with ubiquitous computing: seven challenges. In: Abowd, Gregory D., Brumitt, B., Shafer, S. (eds.) UbiComp 2001. LNCS, vol. 2201, pp. 256–272. Springer, Heidelberg (2001). doi:10.1007/3-540-45427-6_22

    Chapter  Google Scholar 

  15. Fox, A., et al.: Integrating information appliances into an interactive workspace. IEEE Comput. Graph. Appl. 20(3), 54–65 (2000)

    Article  MathSciNet  Google Scholar 

  16. Gama, K., et al.: Combining heterogeneous service technologies for building an Internet of Things middleware. Comput. Commun. 35(4), 405–417 (2012)

    Article  Google Scholar 

  17. González García, C., et al.: Midgar: generation of heterogeneous objects interconnecting applications. a domain specific language proposal for Internet of Things scenarios. Comput. Netw. 64, 143–158 (2014)

    Google Scholar 

  18. Grabler, F., et al.: Generating photo manipulation tutorials by demonstration. In: ACM SIGGRAPH 2009 Papers. pp. 66:1–66:9. ACM, New York (2009)

    Google Scholar 

  19. Greenberg, S.: Context as a dynamic construct. Hum.-Comput. Interact. 16(2), 257–268 (2001)

    Article  Google Scholar 

  20. Guinard, D., et al.: Towards physical mashups in the web of things. In: 2009 Sixth International Conference on Networked Sensing Systems (INSS), pp. 1–4. IEEE (2009)

    Google Scholar 

  21. Guinard, D., Trifa, V.: Towards the web of things: web mashups for embedded devices. In: Workshop on Mashups, Enterprise Mashups and Lightweight Composition on the Web (MEM 2009), in Proceedings of WWW (International World Wide Web Conferences), Madrid, Spain (2009)

    Google Scholar 

  22. Humble, J., Crabtree, A., Hemmings, T., Åkesson, K.-P., Koleva, B., Rodden, T., Hansson, P.: “Playing with the bits” user-configuration of ubiquitous domestic environments. In: Dey, Anind K., Schmidt, A., McCarthy, Joseph F. (eds.) UbiComp 2003. LNCS, vol. 2864, pp. 256–263. Springer, Heidelberg (2003). doi:10.1007/978-3-540-39653-6_20

    Chapter  Google Scholar 

  23. Jesdabodi, C., Maalej, W.: Understanding usage states on mobile devices. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 1221–1225. ACM, New York (2015)

    Google Scholar 

  24. Katasonov, A., et al.: Smart semantic middleware for the Internet of Things. ICINCO-ICSO 8, 169–178 (2008)

    Google Scholar 

  25. Kubitza, T., Schmidt, A.: Towards a toolkit for the rapid creation of smart environments. In: Díaz, P., Pipek, V., Ardito, C., Jensen, C., Aedo, I., Boden, A. (eds.) IS-EUD 2015. LNCS, vol. 9083, pp. 230–235. Springer, Cham (2015). doi:10.1007/978-3-319-18425-8_21

    Chapter  Google Scholar 

  26. Lee, T.Y., et al.: Towards understanding human mistakes of programming by example: an online user study. In: Proceedings of the 22nd International Conference on Intelligent User Interfaces, pp. 257–261. ACM, New York (2017)

    Google Scholar 

  27. Leshed, G., et al.: CoScripter: automating and sharing how-to knowledge in the enterprise. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1719–1728. ACM, New York (2008)

    Google Scholar 

  28. Li, T.J.-J., et al.: Designing a conversational interface for a multimodal smartphone programming-by-demonstration agent. In: Conversational UX Design CHI 2017 Workshop, Denver, CO (2017, in press)

    Google Scholar 

  29. Li, T.J.-J., et al.: SUGILITE: creating multimodal smartphone automation by demonstration. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, Denver (2017, in press)

    Google Scholar 

  30. Lieberman, H.: Your Wish is My Command: Programming by Example. Morgan Kaufmann, San Francisco (2001)

    Google Scholar 

  31. McDaniel, R.G., Myers, B.A.: Gamut: demonstrating whole applications. In: Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology, pp. 81–82 ACM, New York (1997)

    Google Scholar 

  32. McDaniel, R.G., Myers, B.A.: Getting more out of programming-by-demonstration. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 442–449 ACM, New York (1999)

    Google Scholar 

  33. Mennicken, S., et al.: From today’s augmented houses to tomorrow’s smart homes: new directions for home automation research. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 105–115. ACM, New York (2014)

    Google Scholar 

  34. Modugno, F., Myers, B.A.: Pursuit: graphically representing programs in a demonstrational visual shell. In: Conference Companion on Human Factors in Computing Systems, pp. 455–456. ACM, New York (1994)

    Google Scholar 

  35. Myers, B.A.: Demonstrational interfaces: a step beyond direct manipulation. Computer 25(8), 61–73 (1992)

    Article  Google Scholar 

  36. Myers, B.A.: Visual programming, programming by example, and program visualization: a taxonomy. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 59–66. ACM, New York (1986)

    Google Scholar 

  37. Pintus, A., et al.: The anatomy of a large scale social web for internet enabled objects. In: Proceedings of the Second International Workshop on Web of Things. pp. 6:1–6:6. ACM, New York (2011)

    Google Scholar 

  38. Raffle, H., et al.: Beyond record and play: backpacks: tangible modulators for kinetic behavior. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 681–690. ACM, New York (2006)

    Google Scholar 

  39. Ricquebourg, V., et al.: The smart home concept: our immediate future. In: 2006 1st IEEE International Conference on E-Learning in Industrial Electronics, pp. 23–28 (2006)

    Google Scholar 

  40. Schmidt, A.: Programming ubiquitous computing environments. In: Díaz, P., Pipek, V., Ardito, C., Jensen, C., Aedo, I., Boden, A. (eds.) IS-EUD 2015. LNCS, vol. 9083, pp. 3–6. Springer, Cham (2015). doi:10.1007/978-3-319-18425-8_1

    Chapter  Google Scholar 

  41. Song, Z., et al.: Semantic middleware for the Internet of Things. In: 2010 Internet of Things (IOT), pp. 1–8 (2010)

    Google Scholar 

  42. Ur, B., et al.: Practical trigger-action programming in the smart home. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 803–812. ACM, New York (2014)

    Google Scholar 

  43. Wiese, J.S.: Evolving the Ecosystem of Personal Behavioral Data (2015)

    Google Scholar 

  44. Zhang, T., Brügge, B.: Empowering the user to build smart home applications. In: Proceedings of 2nd International Conference on Smart Homes and Health Telematics (ICOST2004), Singapore (2004)

    Google Scholar 

  45. Zhao, S., et al.: Discovering different kinds of smartphone users through their application usage behaviors. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 498–509. ACM, New York (2016)

    Google Scholar 

  46. Zhong, Y., et al.: Smart home on smart phone. In: Proceedings of the 13th International Conference on Ubiquitous Computing, pp. 467–468. ACM, New York (2011)

    Google Scholar 

Download references

Acknowledgement

This work was supported in part by the Yahoo! InMind project.

Author information

Authors and Affiliations

  1. Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, USA

    Toby Jia-Jun Li, Fanglin Chen & Brad A. Myers

  2. School of Electronics Engineering and Computer Science, Peking University, Beijing, China

    Yuanchun Li

Authors
  1. Toby Jia-Jun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuanchun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fanglin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Brad A. Myers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Toby Jia-Jun Li or Brad A. Myers .

Editor information

Editors and Affiliations

  1. Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro, Brazil

    Simone Barbosa

  2. Eindhoven University of Technology, Eindhoven, The Netherlands

    Panos Markopoulos

  3. C.N.R. - ISTI , Pisa, Italy

    Fabio Paternò

  4. City, University of London, London, United Kingdom

    Simone Stumpf

  5. Università degli Studi di Milano, Milan, Italy

    Stefano Valtolina

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Li, T.JJ., Li, Y., Chen, F., Myers, B.A. (2017). Programming IoT Devices by Demonstration Using Mobile Apps. In: Barbosa, S., Markopoulos, P., Paternò, F., Stumpf, S., Valtolina, S. (eds) End-User Development. IS-EUD 2017. Lecture Notes in Computer Science(), vol 10303. Springer, Cham. https://doi.org/10.1007/978-3-319-58735-6_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-58735-6_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-58734-9

  • Online ISBN: 978-3-319-58735-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation