Advertisement

The Interaction Engine

  • Nikolas Martelaro
  • Wendy Ju
  • Mark Horowitz
Chapter
Part of the Understanding Innovation book series (UNDINNO)

Abstract

The Interaction Engine is a framework for prototyping interactive, connected devices based on widely available single-board Linux computers. With microcontrollers, networking, and modular open-source software, these modules enable interaction modalities such as audio, video, tangible, and digital interfaces to be embedded into forms that go beyond traditional computing. In this paper, we outline the hardware and software components that make up the general Interaction Engine framework and discuss its benefits for interaction designers. We provide an illustrative case study of the Interaction Engine in use. We ran workshops to introduce designers to the Interaction Engine framework and we describe the projects where they subsequently employed Interaction Engines to understand issues and opportunities presented by this model. In describing the framework and case studies, we intend to shift designer’s thinking of computer as product to computer as material to create new interactive devices.

Notes

Acknowledgements

We would like to thank the following people for their help with this work: Michael Shiloh, Victor Chahuneau, Tom Igoe, Bjoern Hartmann, Rob Semmens, Dickson Chow, Phillip Dupree, Noam Zomerfeld, Carey Smith, Nathan Seidel, Bill Verplank, CCA & the Stanford Design Group.

References

  1. Acholonu, U. C. (2012). Techonology [sic] bricolage exploring the tension between convention and innovation when problem-solving with technology. Stanford University.Google Scholar
  2. Adafruit. (2016). Adafruit-occidentalis. Retrieved August 1, 2016, from https://github.com/adafruit/Adafruit-Occidentalis
  3. Amazon. (2016a). Amazon dash button. Retrieved July 31, 2016, from https://www.amazon.com/Dash-Buttons/b?ie=UTF8&node=10667898011
  4. Amazon. (2016b). Amazon echo. Retrieved July 30, 2016, from https://www.amazon.com/Amazon-Echo-Bluetooth-Speaker-with-WiFi-Alexa/dp/B00X4WHP5E
  5. Aminzade, D., Hartmann, B., Doorley, S., et al. (2008). Interactive digital murals at San Jose Museum of Art. Google Scholar
  6. ARM. (2016). Cortex-M Series. Retrieved July 31, 2016, from http://www.arm.com/products/processors/cortex-m/index.php
  7. Ballagas, R., Ringel, M., Stone, M., & Borchers, J. (2003). iStuff: A physical user interface toolkit for ubiquitous computing environments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 537–544). New York: ACM.Google Scholar
  8. Banzi, M., & Shiloh, M. (2014). Make: Getting started with Arduino: The open source electronics prototyping platform. Sebastopol: Maker Media.Google Scholar
  9. Barragán, H. (2004). Wiring: Prototyping physical interaction design.Google Scholar
  10. Beigl, M., Gellersen, H.-W., & Schmidt, A. (2001). Mediacups: Experience with design and use of computer-augmented everyday artefacts. Computer Networks, 35, 401–409. doi: 10.1016/S1389-1286(00)00180-8.CrossRefGoogle Scholar
  11. Berdahl, E., & Ju, W. (2011). Satellite CCRMA: A musical interaction and sound synthesis platform. In Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 173–178).Google Scholar
  12. Borthwick, M. (2016). Building museum exhibits with open hardware: Raspberry Pi & Arduino.Google Scholar
  13. Bostock, M. D3.js—Data-driven documents. Retrieved July 26, 2016, from https://d3js.org/
  14. Buechley, L., Eisenberg, M., Catchen, J., & Crockett, A. (2008) The LilyPad Arduino: Using computational textiles to investigate engagement, aesthetics, and diversity in computer science education. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 423–432). New York: ACM.Google Scholar
  15. Buechley, L., Elumeze, N., & Eisenberg, M. (2006). Electronic/computational textiles and children’s crafts. In Proceedings of the 2006 Conference on Interaction Design and Children (pp. 49–56). New York: ACM.Google Scholar
  16. Buechley, L., & Hill, B. M. (2010). LilyPad in the wild: How hardware’s long tail is supporting new engineering and design communities. In Proceedings of the 8th ACM Conference on Designing Interactive Systems (pp. 199–207). New York: ACM.Google Scholar
  17. Bush, V. (1989). As we may think. In Perspectives on the computer revolution (p. 49).Google Scholar
  18. Chan, L., Hsieh, C.-H., Chen, Y.-L., Yang, S., Huang, D.-Y., Liang, R.-H., et al. (2015) Cyclops: Wearable and single-piece full-body gesture input devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 3001–3009). New York: ACM.Google Scholar
  19. Charlesworth, T., Ford, H., Milton, L., Mortensson, T., Pedlingham, J., Knibbe, J., et al. (2015). TellTale: Adding a polygraph to everyday life. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (pp. 1693–1698). New York: ACM.Google Scholar
  20. Chung, E. S., Hong, J. I., Lin, J., Prabaker, M. K., & Landay, J. A. (2004). Development and evaluation of emerging design patterns for ubiquitous computing. In Proceedings of the 5th conference on designing interactive systems: processes, practices, methods, and techniques (pp. 233–242). New York: ACM.Google Scholar
  21. Derry, S. (1990). Flexible cognitive tools for problem solving instruction. In Annual meeting of the American Educational Research Association, Boston, MA.Google Scholar
  22. DiCola, T. (2015). Setting up WiFi with BeagleBone Black|Adafruit Learning System. Retrieved July 30, 2016, from https://learn.adafruit.com/setting-up-wifi-with-beaglebone-black/overview
  23. Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwarz, D. L., & Klemmer, S. R. (2010). Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Transactions on Computer-Human Interaction TOCHI, 17, 18. doi: 10.1145/1879831.1879836.Google Scholar
  24. Electric Imp, Inc. (2016). Electric Imp—Electric imp hardware. Retrieved July 31, 2016, from https://www.electricimp.com/platform/hardware/
  25. Fitbit. (2016). Fitbit official site. Retrieved July 31, 2016, from https://www.fitbit.com/
  26. Frei, P., Su, V., Mikhak, B., & Ishii, H. (2000). Curlybot: Designing a new class of computational toys. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 129–136). New York: ACM.Google Scholar
  27. Gaver, W., Michael, M., Kerridge, T., Wilkie, A., Boucher, A., Ovalle, L., et al. (2015). Energy babble: Mixing environmentally-oriented internet content to engage community groups. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 1115–1124). New York: ACM.Google Scholar
  28. Gibb, A. (2009). Bug labs: Hacks and apps. Linux J, 2009, 4.Google Scholar
  29. Gibb, A. M. (2010). New media art, design, and the Arduino microcontroller: A malleable tool. Pratt Institute.Google Scholar
  30. Greenberg, S., & Fitchett, C. (2001). Phidgets: Easy development of physical interfaces through physical widgets. In Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology (pp. 209–218). New York: ACM.Google Scholar
  31. Hardkernal. (2016a). ODROID U3. Retrieved July 30, 2016, from http://www.hardkernel.com/main/products/prdt_info.php?g_code=g138745696275
  32. Hardkernal. (2016b). ODROID C1. Retrieved July 30, 2016, from http://www.hardkernel.com/main/products/prdt_info.php?g_code=G141578608433
  33. Harris-Cronin, S. (2008). High resolution video playback.Google Scholar
  34. Hartmann, B., Klemmer, S. R., Bernstein, M., Abdulla, L., Burr, B., Robinson-Mosher, A., et al. (2006). Reflective physical prototyping through integrated design, test, and analysis. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (pp. 299–308). New York: ACM.Google Scholar
  35. Hartmann, B., & Wright, P. K. (2013). Designing bespoke interactive devices. Computer, 46, 85–89. doi: 10.1109/MC.2013.274.CrossRefGoogle Scholar
  36. Hennigh-Palermo, S. G. (2015). p5bots. Retrieved July 21, 2016, from https://github.com/sarahgp/p5bots
  37. Huang, A. B. (2008). Chumby: An experiment in hackable pervasive computing. IEEE Pervasive Computing, 7, 55–62.CrossRefGoogle Scholar
  38. IBM. (2016) watson-developer-cloud/text-to-speech-nodejs. Retrieved July 26, 2016, fromhttps://github.com/watson-developer-cloud/text-to-speech-nodejs
  39. Igoe, T. (2014). Lab: Serial communication with Node.js—ITP physical computing.Google Scholar
  40. Itseez. (2016). OpenCV|OpenCV. Retrieved July 26, 2016, from http://opencv.org/
  41. Jibo, Inc. (2016). Jibo—The worlds first social robot. Retrieved July 31, 2016, from https://www.jibo.com/
  42. Jonassen, D. H. (1994). Technology as cognitive tools: Learners as designers. ITForum Paper, 1, 67–80.Google Scholar
  43. Kaufmann, B., & Buechley, L. (2010). Amarino: A toolkit for the rapid prototyping of mobile ubiquitous computing. In: Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services (pp. 291–298). New York: ACM.Google Scholar
  44. Kay, A. (1998). Alan Kay on messaging.Google Scholar
  45. Kubitza, T., Schmidt, A., Pohl, N., Petrelli, D., Dingler, T., & Dulake, N. (2013). Tools and methods for creating interactive artifacts. In Proceedings of the 8th International Conference on Tangible, Embedded and Embodied Interaction (pp. 385–388). New York: ACM.Google Scholar
  46. Kuniavsky, M. (2010). Smart things: Ubiquitous computing user experience design. Amsterdam: Elsevier.Google Scholar
  47. Landin, H. (2005). Fragile and magical: Materiality of computational technology as design material. In Proceedings of the 4th Decennial Conference on Critical Computing: Between Sense and Sensibility (pp. 117–120). New York: ACM.Google Scholar
  48. Langer, M., & Alderman, J. (2016). Open hardware belongs in your museum. In Museums and the Web 2016. Los Angeles, CA.Google Scholar
  49. Lee, J. C., Avrahami, D., Hudson, S. E., Forlizzi, J., Dietz, P. H., & Leigh, D. (2004). The Calder Toolkit: wired and wireless components for rapidly prototyping interactive devices. In Proceedings of the 5th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (pp. 167–175). New York: ACM.Google Scholar
  50. Levis, P., Madden, S., Polastre, J., Szewczyk, R., Whitehouse, K., Woo, A., et al. (2005). TinyOS: An operating system for sensor networks. In: W. Weber, J. M. Rabaey, E. Aarts (Eds.), Ambient intelligence (pp. 115–148). Berlin: Springer.Google Scholar
  51. Lin, F. X., Rahmati, A., & Zhong, L. (2010). Dandelion: A framework for transparently programming phone-centered wireless body sensor applications for health. In Wireless health 2010 (pp. 74–83). New York: ACM.CrossRefGoogle Scholar
  52. Marshall, M. T., Dulake, N., Petrelli, D., & Kockelkorn, H. (2015). From the deposit to the exhibit floor: An exploration on giving museum objects personality and social life. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (pp. 1917–1922). New York: ACM.Google Scholar
  53. Martelaro, N., Shiloh, M., & Ju, W. (2016). The interaction engine: Tools for prototyping connected devices. In Proceedings of the TEI ‘16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction (pp. 762–765). New York: ACM.Google Scholar
  54. Marvell. (2016). Kinoma Create. Retrieved July 31, 2016, from http://kinoma.com/create/
  55. mBed. (2016). Nordic nRF51822. Retrieved July 31, 2016, from https://developer.mbed.org/platforms/Nordic-nRF51822/
  56. McCarthy, L. (2016). p5.js (Computer Program).Google Scholar
  57. McComb, G. (2003). Robot builder’s bonanza (3rd ed.). New York: McGraw-Hill.Google Scholar
  58. McGrath, W., Etemadi, M., Roy, S., & Hartmann, B. (2015). Fabryq: Using phones as gateways to prototype internet of things applications using web scripting. In Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (pp. 164–173). New York: ACM.Google Scholar
  59. Mellis, D., Banzi, M., Cuartielles, D., & Igoe, T. (2007). Arduino: An open electronic prototyping platform. In Proceedings of CHI.Google Scholar
  60. Mellis, D. A., & Buechley, L. (2014). Do-it-yourself cellphones: An investigation into the possibilities and limits of high-tech diy. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1723–1732). New York: ACM.Google Scholar
  61. Mellis, D. A, Jacoby, S., Buechley, L., Perner-Wilson, H., & Qi, J. (2013). Microcontrollers as material: Crafting circuits with paper, conductive ink, electronic components, and an “untoolkit.” In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction (pp. 83–90). New York: ACM.Google Scholar
  62. Merrill, D., Kalanithi, J., & Maes, P. (2007). Siftables: Towards sensor network user interfaces. In: Proceedings of the 1st International Conference on Tangible and Embedded Interaction (pp. 75–78). New York: ACM.Google Scholar
  63. Millner, A., & Baafi, E. (2011). Modkit: Blending and extending approachable platforms for creating computer programs and interactive objects. In Proceedings of the 10th International Conference on Interaction Design and Children (pp. 250–253). New York: ACM.Google Scholar
  64. Norman, D. A. (2013). The design of everyday things: Revised and expanded edition. New York: Basic books.Google Scholar
  65. O’Sullivan, D., & Igoe, T. (2004). Physical computing: Sensing and controlling the physical world with computers. Boston, MA: Course Technology Press.Google Scholar
  66. Output Arts. (2012). I have a message for you…Google Scholar
  67. Overholt, D., Møbius, N. “DZL”. (2013). Embedded audio without beeps: Synthesis and sound effects from cheap to steep. In Proceedings of the 8th International Conference on Tangible, Embedded and Embodied Interaction (pp. 361–364). New York: ACM.Google Scholar
  68. Palazzetti, E. (2015). Getting Started with UDOO. Birmingham: Packt.Google Scholar
  69. Punch Through Design. (2016). LightBlue Bean. Retrieved July 31, 2016, from https://punchthrough.com/bean
  70. Qi, J., & Buechley, L. (2010). Electronic popables: Exploring paper-based computing through an interactive pop-up book. In Proceedings of the Fourth International Conference on Tangible, Embedded, and Embodied Interaction (pp. 121–128). New York: ACM.CrossRefGoogle Scholar
  71. Raghavan, P., Lad, A., & Neelakandan, S. (2005). Embedded linux system design and development. Boca Raton: CRC Press.CrossRefGoogle Scholar
  72. Redmer, A., & Stoeveken, T. (2016). MJPEG-STREAMER.Google Scholar
  73. Resnick, M., Martin, F., Berg, R., Borovoy, R., Colella, V., Kramer, K., et al. (1998). Digital manipulatives: New toys to think with. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 281–287). New York: ACM Press/Addison-Wesley.Google Scholar
  74. Riliskis, L., & Levis, P. (2014). Ravel a framework for embedded-gateway-cloud applications. In Proceedings of the 12th ACM Conference on Embedded Network Sensor Systems (pp. 344–345). New York: ACM.Google Scholar
  75. Roman, M., Al-Muhtadi, J., Ziebart, B., Campbell, R., & Mickunas, M. D. (2003). System support for rapid ubiquitous computing application development and evaluation. In System Support for Ubiquitous Computing Workshop (UbiSys’ 03) in conjunction with UbiComp.Google Scholar
  76. Rozin, D. (1999). Wooden mirror.Google Scholar
  77. Russell, S. C. (2012). Pi and Arduino in action. The MagPi, 4–6.Google Scholar
  78. Sculley, J. (1987). Odyssey: Pepsi to apple, a journey of adventure, ideas, and the future. New York: Harper & Row.Google Scholar
  79. Sheppard, S. D. (1992). Mechanical dissection: An experience in how things work. In Proceedings of the Engineering Education: Curriculum Innovation & Integration, January 6–10.Google Scholar
  80. Sipitakiat, A., & Blikstein, P. (2013). Interaction design and physical computing in the era of miniature embedded computers. In Proceedings of the 12th International Conference on Interaction Design and Children (pp. 515–518). New York: ACM.CrossRefGoogle Scholar
  81. Spark Labs, Inc. (2016) Particle. In Particle. Retrieved July 31, 2016, from https://www.particle.io/prototype
  82. Stankovic, J. A., Lee, I., Mok, A., & Rajkumar, R. (2005). Opportunities and obligations for physical computing systems. Computer, 38, 23–31. doi: 10.1109/MC.2005.386.CrossRefGoogle Scholar
  83. Stiller, E. (2009). Teaching programming using bricolage. Journal of Computing Sciences in Colleges, 24, 35–42.Google Scholar
  84. Tahiroğlu, K., Svedström, T., & Wikström, V. (2015). NOISA: A novel intelligent system facilitating smart interaction. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (pp. 279–282). New York: ACM.Google Scholar
  85. Tan, S. L., & Nguyen, B. A. T. (2009). Survey and performance evaluation of real-time operating systems (RTOS) for small microcontrollers. IEEE Micro. doi:  10.1109/MM.2009.56.
  86. Technical io. (2016). Tessel. Retrieved July 31, 2016, from https://tessel.io/
  87. Texas Instruments. (2016). Simplelink SensorTag.Google Scholar
  88. The Workers. (2015). After dark (Artwork).Google Scholar
  89. Townsend, A. M. (2013). Smart cities: Big data, civic hackers, and the quest for a new utopia. New York: WW Norton & Company.Google Scholar
  90. Tullio, J., Dey, A. K., Chalecki, J., & Fogarty, J. (2007). How it works: A field study of non-technical users interacting with an intelligent system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 31–40). New York: ACM.CrossRefGoogle Scholar
  91. Vallgårda, A., & Redström, J. (2007). Computational composites. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 513–522). New York: ACM.CrossRefGoogle Scholar
  92. Vallgårda, A., & Sokoler, T. A. Material strategy: Exploring material properties of computers. International Journal of Design. Retrieved August 2, 2016, from http://www.ijdesign.org/ojs/index.php/IJDesign/article/view/628/309
  93. Van Every, S. (2015). p5.serialport (Computer Program). Retrieved July 21, 2016, from https://github.com/vanevery/p5.serialport
  94. Vandevelde, C., Vanhoucke, M., & Saldien, J. (2015). Prototyping social interactions with DIY animatronic creatures. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (pp. 509–512). New York: ACM.Google Scholar
  95. Villar, N., Scott, J., Hodges, S., Hammil, K., & Miler, C. (2012). .NET gadgeteer: A platform for custom devices. In Pervasive computing (pp. 216–233). Berlin: Springer.Google Scholar
  96. Waldron, R. (2012). rwaldron/johnny-five (Computer Program). Retrieved July 21, 2016, from https://github.com/rwaldron/johnny-five
  97. Weiser, M. (1999). The computer for the 21st century. SIGMOBILE Mobile Computing and Communications Review, 3, 3–11. doi: 10.1145/329124.329126.CrossRefGoogle Scholar
  98. Willow Garage. (2011). Turtlebot. In Turtlebot. Retrieved January 4, 2017, from http://www.turtlebot.com
  99. Wilson, S., Gurevich, M., Verplank, B., & Stang, P. (2003). Microcontrollers in music HCI instruction: Reflections on our switch to the Atmel AVR platform. In Proceedings of the 2003 Conference on New Interfaces for Musical Expression (pp. 24–29). Singapore: National University of Singapore.Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Center for Design ResearchStanfordUSA

Personalised recommendations