Abstract
As social distancing is becoming the new normal, technology holds the potential to bridge this societal gap through novel interaction modalities that allow multiple users to collaborate and create content together. We present Jammify, an interactive multi-sensory system that focuses on providing a unique digital art-jamming experience with a visual display and a wearable arm-sleeve. The ‘jamming-canvas’ visual display is a two-sided LED light wall (2 m \(\times \) 6 m) where users can draw free-hand gestures on either side and switch between two view modes: own-view and shared-view. The arm-sleeve uses shape-memory-alloy integrated fabric to sense and re-create a subtle and natural touch sensation on each other’s hands. We describe the details of the design and interaction possibilities based on the diverse combinations of both input and output modalities of the system, as well as findings from a user study with ten participants.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ackad, C., Tomitsch, M., Kay, J.: Skeletons and silhouettes: comparing user representations at a gesture-based large display. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI 2016, pp. 2343–2347. Association for Computing Machinery, New York (2016). https://doi.org/10.1145/2858036.2858427
Akshita, Alagarai Sampath, H., Indurkhya, B., Lee, E., Bae, Y.: Towards multimodal affective feedback: interaction between visual and haptic modalities. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI 2015, pp. 2043–2052. Association for Computing Machinery, New York (2015). https://doi.org/10.1145/2702123.2702288
Ananny, M., Strohecker, C., Biddick, K.: Shifting scales on common ground: developing personal expressions and public opinions. Int. J. Continuing Eng. Educ. Life Long Learn. 14(6), 484–505 (2004)
Baer, M., Leenders, R.T.A., Oldham, G.R., Vadera, A.K.: Win or lose the battle for creativity: the power and perils of intergroup competition. Acad. Manag. J. 53(4), 827–845 (2010)
Bakdash, J.Z., Augustyn, J.S., Proffitt, D.R.: Large displays enhance spatial knowledge of a virtual environment. In: Proceedings of the 3rd Symposium on Applied Perception in Graphics and Visualization, APGV 2006, pp. 59–62. Association for Computing Machinery, New York (2006). https://doi.org/10.1145/1140491.1140503
Barthelmess, P., Kaiser, E., Lunsford, R., McGee, D., Cohen, P., Oviatt, S.: Human-centered collaborative interaction. In: Proceedings of the 1st ACM International Workshop on Human-Centered Multimedia, HCM 2006, pp. 1–8. Association for Computing Machinery, New York (2006). https://doi.org/10.1145/1178745.1178747
Boehner, K., DePaula, R., Dourish, P., Sengers, P.: Affect: from information to interaction. In: Proceedings of the 4th Decennial Conference on Critical Computing: Between Sense and Sensibility, CC 2005, pp. 59–68. Association for Computing Machinery, New York (2005). https://doi.org/10.1145/1094562.1094570
Bonanni, L., Vaucelle, C.: Affective touchcasting. In: ACM SIGGRAPH 2006 Sketches, SIGGRAPH 2006, p. 35-es. Association for Computing Machinery, New York (2006). https://doi.org/10.1145/1179849.1179893
Brave, S., Dahley, A.: Intouch: a medium for haptic interpersonal communication. In: Extended Abstracts on Human Factors in Computing Systems, CHI 1997, CHI EA 1997, pp. 363–364. Association for Computing Machinery, New York (1997). https://doi.org/10.1145/1120212.1120435
Brignull, H., Rogers, Y.: Enticing people to interact with large public displays in public spaces. In: Proceedings of INTERACT, Brighton, UK, vol. 3, pp. 17–24 (2003)
Brown, A.R., Sorensen, A.: Dynamic media arts programming in impromptu. In: Proceedings of the 6th ACM SIGCHI Conference on Creativity & Cognition, pp. 245–246 (2007)
Burkitt, E., Barrett, M., Davis, A.: Effects of different emotion terms on the size and colour of children’s drawings. Int. J. Art Ther. 14(2), 74–84 (2009)
Depth Cameras Intel RealSense (2020). https://www.intelrealsense.com/depth-camera-d435/. Accessed 10 Apr 2020
Cheok, A.D., et al.: BlogWall: social and cultural interaction for children. Adv. Hum.-Comput. Interact. 2008 (2008)
Churchill, E.F., Nelson, L., Denoue, L., Helfman, J., Murphy, P.: Sharing multimedia content with interactive public displays: a case study. In: Proceedings of the 5th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, pp. 7–16 (2004)
Coelho, M., Maes, P.: Sprout I/O: a texturally rich interface. In: Proceedings of the 2nd International Conference on Tangible and Embedded Interaction, TEI 2008, pp. 221–222. Association for Computing Machinery, New York (2008). https://doi.org/10.1145/1347390.1347440
Czerwinski, M., Robertson, G., Meyers, B., Smith, G., Robbins, D., Tan, D.: Large display research overview. In: Extended Abstracts on Human Factors in Computing Systems, CHI 2006, CHI EA 2006, pp. 69–74. Association for Computing Machinery, New York (2006). https://doi.org/10.1145/1125451.1125471
Czerwinski, M., Smith, G., Regan, T., Meyers, B., Robertson, G.G., Starkweather, G.K.: Toward characterizing the productivity benefits of very large displays. In: Interact, vol. 3, pp. 9–16 (2003)
Davis, N., Hsiao, C.P., Singh, K.Y., Lin, B., Magerko, B.: Creative sense-making: quantifying interaction dynamics in co-creation. In: Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition, C&C 2017, pp. 356–366. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3059454.3059478
Duarte, L., Carriço, L.: The collaboration platform: a cooperative work course case-study. In: 2010 Eighth International Conference on Creating, Connecting and Collaborating through Computing, pp. 19–25. IEEE (2010)
Eid, M.A., Al Osman, H.: Affective haptics: current research and future directions. IEEE Access 4, 26–40 (2015)
Eisenberg, J., Thompson, W.F.: The effects of competition on improvisers’ motivation, stress, and creative performance. Creat. Res. J. 23(2), 129–136 (2011)
Elvitigala, S., Chan, S.W.T., Howell, N., Matthies, D.J., Nanayakkara, S.: Doodle daydream: an interactive display to support playful and creative interactions between co-workers. In: Proceedings of the Symposium on Spatial User Interaction, SUI 2018, p. 186. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3267782.3274681
Fan, J.E., Dinculescu, M., Ha, D.: Collabdraw: an environment for collaborative sketching with an artificial agent. In: Proceedings of the 2019 on Creativity and Cognition, pp. 556–561. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3325480.3326578
Fan, L., Yu, C., Shi, Y.: Guided social sharing of emotions through drawing art therapy: generation of deep emotional expression and helpful emotional responses. In: Proceedings of the Seventh International Symposium of Chinese CHI, Chinese CHI 2019, pp. 65–78. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3332169.3333571
Felix, E.R., Galoian, K.A., Aarons, C., Brown, M.D., Kearing, S.A., Heiss, U.: Utility of quantitative computerized pain drawings in a sample of spinal stenosis patients. Pain Med. 11(3), 382–389 (2010)
Ghinea, G., Gill, D., Frank, A., De Souza, L.: Using geographical information systems for management of back-pain data. J. Manag. Med. (2002)
Gupta, A., Irudayaraj, A.A.R., Balakrishnan, R.: HapticClench: investigating squeeze sensations using memory alloys. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST 2017, pp. 109–117. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3126594.3126598
Gupta, A., Irudayaraj, A.A.R., Balakrishnan, R.: HapticClench: investigating squeeze sensations using memory alloys. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 109–117 (2017)
Gutwin, C., Greenberg, S.: A descriptive framework of workspace awareness for real-time groupware. Comput. Supported Cooper. Work (CSCW) 11(3–4), 411–446 (2002)
Hamdan, N.A.H., Wagner, A., Voelker, S., Steimle, J., Borchers, J.: Springlets: expressive, flexible and silent on-skin tactile interfaces. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3290605.3300718
Harrison, C., Horstman, J., Hsieh, G., Hudson, S.: Unlocking the expressivity of point lights. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2012, pp. 1683–1692. Association for Computing Machinery, New York (2012). https://doi.org/10.1145/2207676.2208296
Hettiarachchi, A., Nanayakkara, S., Yeo, K.P., Shilkrot, R., Maes, P.: Fingerdraw: more than a digital paintbrush. In: Proceedings of the 4th Augmented Human International Conference, AH 2013, pp. 1–4. Association for Computing Machinery, New York (2013). https://doi.org/10.1145/2459236.2459237
Huber, J., Shilkrot, R., Maes, P., Nanayakkara, S.: Assistive Augmentation, 1st edn. Springer, Heidelberg (2017)
Ilievski, F., Mazzeo, A.D., Shepherd, R.F., Chen, X., Whitesides, G.M.: Soft robotics for chemists. Angew. Chem. Int. Ed. 50(8), 1890–1895 (2011)
Indurkhya, B.: Towards multimodal affective stimulation: interaction between visual, auditory and haptic modalities. In: Stojanov, G., Kulakov, A. (eds.) ICT Innovations 2016. AISC, vol. 665, pp. 3–8. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-68855-8_1
Jang, A., MacLean, D.L., Heer, J.: BodyDiagrams: improving communication of pain symptoms through drawing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2014, pp. 1153–1162. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2556288.2557223
Journeaux, J., Gorrill, H. (eds.): Collective and Collaborative Drawing in Contemporary Practice. Cambridge Scholars Publishing, UK (2017)
Kettebekov, S., Sharma, R.: Toward natural gesture/speech control of a large display. In: Little, M.R., Nigay, L. (eds.) EHCI 2001. LNCS, vol. 2254, pp. 221–234. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-45348-2_20
Khamis, M., Hoesl, A., Klimczak, A., Reiss, M., Alt, F., Bulling, A.: EyeScout: active eye tracking for position and movement independent gaze interaction with large public displays. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST 2017, pp. 155–166. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3126594.3126630
Kheirikhah, M.M., Rabiee, S., Edalat, M.E.: A review of shape memory alloy actuators in robotics. In: Ruiz-del-Solar, J., Chown, E., Plöger, P.G. (eds.) RoboCup 2010. LNCS (LNAI), vol. 6556, pp. 206–217. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20217-9_18
Knoop, E., Rossiter, J.: The tickler: a compliant wearable tactile display for stroking and tickling. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA 2015, pp. 1133–1138. Association for Computing Machinery, New York (2015). https://doi.org/10.1145/2702613.2732749
Knoop, E., Rossiter, J.: The tickler: a compliant wearable tactile display for stroking and tickling. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1133–1138 (2015)
Leahu, L., Schwenk, S., Sengers, P.: Subjective objectivity: negotiating emotional meaning. In: Proceedings of the 7th ACM Conference on Designing Interactive Systems, pp. 425–434 (2008)
Martin, K., Penn, A., Gavin, L.: Engaging with a situated display via picture messaging. In: Extended Abstracts on Human Factors in Computing Systems, CHI 2006, pp. 1079–1084 (2006)
McCarthy, J.: Using public displays to create conversation opportunities. In: Proceedings of Workshop on Public, Community, and Situated Displays at CSCW 2002 (2002)
Michelis, D., Müller, J.: The audience funnel: observations of gesture based interaction with multiple large displays in a city center. Int. J. Hum.-Comput. Interact. 27(6), 562–579 (2011)
Moon, B.L.: The Role of Metaphor in Art Therapy: Theory, Method, and Experience. Charles C Thomas Publisher (2007)
Mueller, F.F., et al.: Next steps for human-computer integration. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI 2020, pp. 1–15. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3313831.3376242
Mullenbach, J., Shultz, C., Colgate, J.E., Piper, A.M.: Exploring affective communication through variable-friction surface haptics. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2014, pp. 3963–3972. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2556288.2557343
Muthukumarana, S., Elvitigala, D.S., Cortes, J.P.F., Matthies, D.J., Nanayakkara, S.: PhantomTouch: creating an extended reality by the illusion of touch using a shape-memory alloy matrix. In: SIGGRAPH Asia 2019 XR, SA 2019, pp. 29–30. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3355355.3361877
Muthukumarana, S., Elvitigala, D.S., Forero Cortes, J.P., Matthies, D.J., Nanayakkara, S.: Touch me gently: recreating the perception of touch using a shape-memory alloy matrix. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI 2020, pp. 1–12. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3313831.3376491
Mynatt, E.D., Igarashi, T., Edwards, W.K., LaMarca, A.: Flatland: new dimensions in office whiteboards. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1999, pp. 346–353. Association for Computing Machinery, New York (1999). https://doi.org/10.1145/302979.303108
Nakao, T., Kunze, K., Isogai, M., Shimizu, S., Pai, Y.S.: FingerFlex: shape memory alloy-based actuation on fingers for kinesthetic haptic feedback. In: 19th International Conference on Mobile and Ubiquitous Multimedia, MUM 2020, pp. 240–244. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3428361.3428404
Nakao, T., et al.: ShareHaptics: a modular haptic feedback system using shape memory alloy for mixed reality shared space applications. In: ACM SIGGRAPH 2019 Posters. SIGGRAPH 2019. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3306214.3338597
Nanayakkara, S., Schroepfer, T., Wyse, L., Lian, A., Withana, A.: SonicSG: from floating to sounding pixels. In: Proceedings of the 8th Augmented Human International Conference, AH 2017. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3041164.3041190
Nanayakkara, S., Taylor, E., Wyse, L., Ong, S.H.: An enhanced musical experience for the deaf: design and evaluation of a music display and a haptic chair. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2009, pp. 337–346. Association for Computing Machinery, New York (2009). https://doi.org/10.1145/1518701.1518756
Nanayakkara, S.C., Wyse, L., Ong, S.H., Taylor, E.A.: Enhancing musical experience for the hearing-impaired using visual and haptic displays. Hum.-Comput. Interact. 28(2), 115–160 (2013). https://doi.org/10.1080/07370024.2012.697006
Nuitrack (2020). https://nuitrack.com/. Accessed 10 Feb 2020
Peng, C.: Survey of collaborative drawing support tools. Comput. Supported Cooper. Work (CSCW) 1(3), 197–228 (1992)
Reetz, A., Gutwin, C., Stach, T., Nacenta, M.A., Subramanian, S.: Superflick: a natural and efficient technique for long-distance object placement on digital tables. Graph. Interface 2006, 163–170 (2006)
Robertson, G., et al.: The large-display user experience. IEEE Comput. Graph. Appl. 25(4), 44–51 (2005)
Rogers, Y., Lindley, S.: Collaborating around vertical and horizontal large interactive displays: which way is best? Interact. Comput. 16(6), 1133–1152 (2004)
Shaballout, N., Neubert, T.A., Boudreau, S., Beissner, F.: From paper to digital applications of the pain drawing: systematic review of methodological milestones. JMIR mHealth uHealth 7(9), e14569 (2019)
Sheridan, J., Bryan-Kinns, N., Reeves, S., Marshall, J., Lane, G.: Graffito: crowd-based performative interaction at festivals. In: Extended Abstracts on Human Factors in Computing Systems, CHI 2011, CHI EA 2011, pp. 1129–1134. Association for Computing Machinery, New York (2011). https://doi.org/10.1145/1979742.1979725
Shoemaker, G., Tang, A., Booth, K.S.: Shadow reaching: a new perspective on interaction for large displays. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, UIST 2007, pp. 53–56. Association for Computing Machinery, New York (2007). https://doi.org/10.1145/1294211.1294221
Suhonen, K., Väänänen-Vainio-Mattila, K., Mäkelä, K.: User experiences and expectations of vibrotactile, thermal and squeeze feedback in interpersonal communication. In: The 26th BCS Conference on Human Computer Interaction 26, pp. 205–214 (2012)
Tang, J.C., Minneman, S.L.: Videodraw: a video interface for collaborative drawing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1990, pp. 313–320. Association for Computing Machinery, New York (1990). https://doi.org/10.1145/97243.97302
Taylor, S., Izadi, S., Kirk, D., Harper, R., Garcia-Mendoza, A.: Turning the tables: an interactive surface for VJing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1251–1254 (2009)
Von Zadow, U.: Using personal devices to facilitate multi-user interaction with large display walls. In: Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology, pp. 25–28 (2015)
Wilson, G., Brewster, S.A.: Multi-moji: combining thermal, vibrotactile and visual stimuli to expand the affective range of feedback. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI 2017, pp. 1743–1755. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3025453.3025614
Yalom, I.D.: The gift of therapy: an open letter to a new generation of therapists and their patients. J. Soc. Work Pract. (2002)
Yeo, K.P., Nanayakkara, S.: SpeechPlay: composing and sharing expressive speech through visually augmented text. In: Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, OzCHI 2013, pp. 565–568. Association for Computing Machinery, New York (2013). https://doi.org/10.1145/2541016.2541061
Yoo, T., Yoo, Y., Choi, S.: An explorative study on crossmodal congruence between visual and tactile icons based on emotional responses. In: Proceedings of the 16th International Conference on Multimodal Interaction, ICMI 2014, pp. 96–103. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2663204.2663231
Zhao, M., Wang, Y., Redmiles, D.: Using collaborative online drawing to build up distributed teams. In: 2017 IEEE 12th International Conference on Global Software Engineering (ICGSE), pp. 61–65. IEEE (2017)
Zhao, M., Wang, Y., Redmiles, D.: Using collaborative online drawing to build up distributed teams. In: 2017 IEEE 12th International Conference on Global Software Engineering (ICGSE), pp. 61–65 (2017). https://doi.org/10.1109/ICGSE.2017.3
Zhao, M., Wang, Y., Redmiles, D.: Using playful drawing to support affective expressions and sharing in distributed teams. In: 2017 IEEE/ACM 2nd International Workshop on Emotion Awareness in Software Engineering (SEmotion), pp. 38–41 (2017). https://doi.org/10.1109/SEmotion.2017.3
Acknowledgement
This work was supported by Assistive Augmentation research grant under the Entrepreneurial Universities (EU) initiative of New Zealand. We thank Peter Cleveland, Robyn Walton, and Prasanth Sasikumar for their assistance in constructing the LED display.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 164393 KB)
Rights and permissions
Copyright information
© 2021 IFIP International Federation for Information Processing
About this paper
Cite this paper
Muthukumarana, S., Elvitigala, D.S., Wu, Q., Pai, Y.S., Nanayakkara, S. (2021). Jammify: Interactive Multi-sensory System for Digital Art Jamming. In: Ardito, C., et al. Human-Computer Interaction – INTERACT 2021. INTERACT 2021. Lecture Notes in Computer Science(), vol 12936. Springer, Cham. https://doi.org/10.1007/978-3-030-85607-6_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-85607-6_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-85606-9
Online ISBN: 978-3-030-85607-6
eBook Packages: Computer ScienceComputer Science (R0)