Skip to main content
SpringerLink
Log in

The Neem Platform: An Evolvable Framework for Perceptual Collaborative Applications

  • Published:
Journal of Intelligent Information Systems Aims and scope Submit manuscript

Abstract

This paper introduces the Neem Platform, a generic research test bed for the development of a novel class of adaptive intelligent collaborative applications. These applications provide support for groups of people working together by facilitating their communication and by reacting adaptively to the perceived contexts of an ongoing interaction. Applications in Neem explore the multimodal context of mostly human-to-human communication, by “overhearing” conversations, monitoring user actions and reacting to perceived opportunities for augmentation via intelligent system interventions.

The Neem Platform is designed to facilitate the integration of functionality for capture and reification of user communicative actions over a variety of modalities (e.g. speech, text, gestures), as well as facilitate analysis and reasoning based on this reified context. The platform promotes an exploratory and evolutionary style of development by affording rapid development of components, enactment of human-controlled real-time (Wizard of Oz) experiments, and remote collection of data for off-line analysis of interactions.

This paper contributes the presentation of a novel coordination mechanism and shows how this underlying mechanism can be used to implement systems that support intelligent reaction to multimodal group interactions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Andreoli, J., Gallaire, H., and Pareschi, R. (1995). Rule-Based Object Coordination. In ObjectBased Models and Languages for Concurrent Systems, volume 924 of Lecture Notes in Computer Science.

  • Banâtre, J.-P. and Le Mètayer, D. (1996). Gamma and the Chemical Reaction Model: Ten Years After. In Coordination Programming: Mechanisms, Models and Semantics.

  • Begeman, M. (2002). Pitfalls of Meeting Electronically. Placeware Forum 2002. Webcast of Executive Presentation, San Francisco, CA.

  • Carriero, N. and Gelernter, D. (1989). Linda in Context. Communications of the ACM, 32(4).

  • Castellani, S. and Ciancarini, P. (1996). Enhancing Coordination and Modularity Mechanisms for a Language with Objects-as-Multisets. In Coordination Languages and Models, volume 1061 of Lecture Notes in Computer Science.

  • Castellani, S., Ciancarini, P., and Rossi, D. (1996). The Shape of Shade: A Coordination System. Technical Report UBLCS 96-5.

  • Chung, Goopeel and Dewan, Prasun. (2001). Flexible Support for Application-Sharing Architecture. In Proceedings of the European CSCW Conference (pp. 99–118).

  • Cohen, M. and Ludwig, F.L. (1991). Multidimensional Audio Window Management. International Journal of Man-Machine Studies, 34.

  • Cohen, P.R., Johnston, M., McGee, D., Smith, I., Oviatt, S., Pittman, J., Chen, L., and Clow, J. (1997). Quickset: Multimodal Interaction for Simulation Set-up and Control. In Proceedings of the Fifth Applied Natural Language Processing meeting.

  • Colorado University Center for Spoken Language Research. Cu communicator page. http://communicator.colorado.edu/.

  • Toshiba Corp. (2004). Bee-gent Multi-Agent Framework. Available on the web at http://www2.toshiba.co.jp/beegent/index.htm.

  • Data Connection. http://www.dataconnection.com.

  • Dewan, Prasun. (1998). Architectures for Collaborative Applications. Trends in Software, special issue on Computer Supported Cooperative Work, 7.

  • Dourish, Paul. (1996). Consistency Guarantees: Exploiting Application Semantics for Consistency Management in a Collaboration Toolkit. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. Concurrency.

  • Dourish, Paul. (1998). Using Metalevel Techniques in a Flexible Toolkit for CSCW Applications. ACM Transactions on Computer-Human Interaction, 5(2).

  • Dourish, Paul and Keith Edwards, W. (2000). A Tale of Two Toolkits: Relating Infrastructure and use in Flexible CSCW Toolkits. Journal on Computer Supported Cooperative Work, 9(1), Special issue on Tailorable Systems and Cooperative Work.

  • Edwards, W. Keith. (1995). Coordination Infrastructure in Collaborative Systems. PhD thesis, Georgia Institute of Technology, College of Computing.

  • Ellis, C. and Gibbs, S. (1989). Concurrency Control in Groupware Systems. In Proceedings of the ACM SIGMOD’89 Conference on the Management of Data.

  • Ellis, C., Gibbs, S., and Rein, G. (1990). Design and Use of a Group Editor. In Engineering for Human Computer Interaction.

  • Elrad, Tzilla, Filman, Robert E., and Bader, Atef. (2001). Aspect-Oriented Programming: Introduction. Communications of the ACM, 44(10), 29–32.

    Article  Google Scholar 

  • Elrad, Tzilla, Filman, Robert E., and Bader, Atef. (Eds). (2001). Communications of the ACM. Special Issue on Aspect-Oriented Programming, 44.

  • Erickson, T., Halverson, C., Kellogg, W.A., Laff, M., and Wolf, T. (2002). Social Translucence: Designing Social Infrastructures that Make Collective Activity Visible. Communications of the ACM—Special issue on Community, ed. J. Preece, 45(4), 40–44.

    Google Scholar 

  • Eugster, Patrick Th., Felber, Pascal A., Guerraoui, Rachid, and Kermarrec, Anne-Marie. (2003). The Many Faces of Publish/Subscribe. ACM Computing Surveys (CSUR), 35(2), 114–131.

    Article  Google Scholar 

  • Graham, John R., Decker, Keith S., and Mersic, Michael. (2003). Decaf—A Flexible Multi Agent System Architecture. Autonomous Agents and Multi-Agent Systems, 7(1/2), 7–27.

    Article  Google Scholar 

  • Greenberg, S. and Roseman, M. (1996). Groupware Toolkits for Synchronous Work. Technical Report 96-589-09, Department of Computer Science, University of Calgary.

  • Greenhalgh, Chris and Benford, Steven. (1995). Massive: A Collaborative Virtual Environment for Teleconferencing. ACM Transactions on Computer-Human Interaction (TOCHI), 2(3), 239–261.

    Article  Google Scholar 

  • Hagsand, Olof. (1996). Interactive Multiuser Ves in the Dive System. IEEE Multimedia, 3(1), 30–39.

    Google Scholar 

  • Haptek. http://www.haptek.com.

  • Hare, A.P., Borgatta, E.F., and Bales, R.F. (Eds). (1955). Small Groups: Studies in Social Interaction. Knopf, New York.

  • Hummes, Jakob, and Merialdo, Bernard. (2000). Design of Extensible Component-Based Groupware. Journal on Computer Supported Cooperative Work, 9(1). Special issue on Tailorable Systems and Cooperative Work.

  • IBM Corporation. IBM Voice Systems. http://www-4.ibm.com/software/speech/.

  • International Telecommunication Union. http://www.itu.org.

  • Isbister, Katherine, Nass, Cliff, Nakanishi, Hideyuki, and Ishida, Toru. (2000). Helper Agent: An Assistant for Human-Human Interaction in a Virtual Meeting Space. In Proceedings of the CHI 2000 Conference,

  • Janus Project. http://www.is.cs.cmu.edu/mie/janus.html.

  • Jebara, Tony, Ivanov, Yuri, Rahimi, Ali, and Pentland, Alex. (2000). Tracking conversational context for machine mediation of human discourse. In AAAI Fall 2000 Symposium—Socially Intelligent Agents—The Human in the Loop.

  • Kang, Seong-Hoon and Kim, Sung-Han. (1996). Realistic Audio Teleconferencing Using Binaural and Auralization Techniques. ETRI Journal, 18(1).

  • Kiczales, Gregor. (1996). Beyond the Black Box: Open Implementation—Soapbox. IEEE Software, 13(1).

  • Koch, Michael and Teege, Gunnar. (1999). Support for Tailoring CSCW Systems: Adaptation by Composition. In Proc. 7th Euromicro Workshop on Parallel and Distributed Processing (pp. 146–152), Funchal, Portugal, February IEEE Press.

  • Labrou, Y., Finin, T., and Peng, Y. (1999). Agent Communication Languages: The Current Land-Scape. IEEE Intelligent Systems, 14(2).

  • Liu, Ying and Plale, Beth. (2003). Survey of Publish Subscribe Event Systems. Technical Report TR574, Indiana University.

  • Low, Colin and Babarit, Laurent. (1998). Distributed 3D Audio Rendering. In International World Wide Web Conference, Brisbane, Australia.

  • Malone, T.W., Lai, K.-Y., and Fry, C. (1995). Experiments with OVAL: A Radically Tailorable Tool for Cooperative Work. ACM Transactions on Information Systems, 13(2), 177–205.

    Article  Google Scholar 

  • Martin, D., Cheyer, A., and Moran, D. (1999). The Open Agent Architecture: A framework for Building Distributed Software Systems. Applied Artificial Intelligence, 13(1/2).

  • McGee, D. and Cohen, P. (2001). Creating Tangible Interfaces by Augmenting Physical Objects with Multimodal Language. In Proceedings of the International Conference on Intelligent User Interfaces (IUI 2001).

  • MITRE Corporation. http://fofoca.mitre.org.

  • Mitre Corporation. (2002). Galaxy Communicator Documentation.

  • Nakanishi, Hideyuki, Yoshida, Chikara, Nishimura, Toshikazu, and Ishida, Toru. (1999). Freewalk: A 3D Virtual Space for Casual Meetings. IEEE MultiMedia, 6(2), 20–28.

    Article  Google Scholar 

  • Nishimoto, Kazushi, Sumi, Yasuyuki, and Mase, Kenji. (1998). Enhancement of Creative Aspects of a Daily Conversation with a Topic Development Agent. In Coordination Technology for Collaborative Applications—Organizations, Processes, and Agents, volume 1364 of Lecture Notes on Computer Science.

  • Nodine, Marian, Fowler, Jerry, Ksiezyk, Tomasz, Perry, Brad, Taylor, Malcolm, and Unruh, Amy. (2000). Active Information Gathering in Infosleuth. International Journal of Cooperative Information Systems, 9(1/2), 3–28.

    Article  Google Scholar 

  • OAA Development Team. Open Agent Architecture: Technical white paper. Technical report, SRI International, 1999.

  • Papadopoulos, George, A., and Arbab, Farhad. (1998). Coordination models and languages. In The Engineering of Large Systems, volume 46 of Advances in Computers. Academic Press.

  • Ramloll, Rameshsharma. (1997). Micis: A multimodal Interface for a Common Information Space. In ECSCW’97 Conference Supplement.

  • Reeves, Byron and Nass, Clifford. (2000). Perceptual user Interfaces: Perceptual Bandwidth. Communications of the ACM, 43(3).

  • Rodenstein, Roy and Donath, Judith S. (2000). Talking in Circles: Designing a Satially-Grounded Audioconferencing Environment. In Proceedings of the CHI 2000 Conference on Human factors in computing systems, (pp. 81–88). ACM Press.

  • Rogina, Ivica and Schaaf, Thomas. (2002). Lecture and Presentation Tracking in an Intelligent Meeting Room. In IEEE International Conference on Multimodal Interfaces—ICMI, (pp. 47), Pittsburgh, Pennsylvania.

  • Roseman, Mark and Greenberg, Saul. (1996). Building Real-Time Groupware with Groupkit, A Groupware Toolkit. ACM Transactions on Computer-Human Interaction, 3(1).

  • Roussev, Vassil, Dewan, Prasun, and Jain, Vibhor. (2000). Composable Collaboration Infrastructures Based on Programming Patterns. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, Component Based Infrastructures.

  • Schmidt, Kjeld and Simone, Carla. (1996). Coordination Mechanisms: Towards a Conceptual Foundation of CSCW systems design. Computer Supported Cooperative Work, 5(2/3), 155–200.

    Article  Google Scholar 

  • Shaumburg, Heike. (2001). Computers as Tools or as Social Actors? The users’ Perspective on Anthropomorphic Agents. International Journal of Cooperative Information Systems, 10(1/2).

  • Stiemerling, Oliver and Cremers, Armin B. 2000. The Evolve Project: Component-Based Tailorability for CSCW applications. AI & Society, 14.

  • Sun, C. and Ellis, C.A. (1998). Operational Transformation in Real Time Group Editors: Issues, Algorithms, and Achievements. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, (pp. 59–68).

  • Sycara, Katia, Paolucci, Massimo, Velsen, Martin Van, and Giampapa, Joseph Andrew. (2003). The RETSINA MAS infrastructure. Special joint issue of Autonomous Agents and MAS, 7(1/2). Also as Carnegie Mellon University’s Tech Report CMU-RI-TR-01-05.

  • Tolksdorf, Robert. (1993). Laura: A Coordination Language for Open Distributed Systems. In International Conference on Distributed Computing Systems.

  • Waibel, Alex. (1996). Interactive Translation of Conversational Speech. IEEE Computer, 29(7).

  • Ward, W. and Pellom, B. (1999). The CU Communicator System. In IEEE Workshop on Automatic Speech Recognition and Understanding.

  • Yamazaki, Yasuhiro and Herder, Jens. (2000). Exploring Spatial Audio Conferencing Functionality in Multiuser Virtual Environments (poster session). In Proceedings of the third international Conference on Collaborative virtual environments (pp. 207–208). ACM Press.

Download references

Author information

Authors and Affiliations

  1. Center for Human-Computer Communication, OGI School of Science & Engineering, Oregon Health & Science University, 20000 NW Walker Road, Beaverton, OR, 97006, USA

    P. Barthelmess

  2. Department of Computer Science, University of Colorado at Boulder, Campus Box 430, Boulder, CO, 80309-0430, USA

    C. A. Ellis

Authors
  1. P. Barthelmess
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. C. A. Ellis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to P. Barthelmess.

Additional information

This work was developed at the University of Colorado at Boulder.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Barthelmess, P., Ellis, C.A. The Neem Platform: An Evolvable Framework for Perceptual Collaborative Applications. J Intell Inf Syst 25, 207–240 (2005). https://doi.org/10.1007/s10844-005-0862-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10844-005-0862-y

Keywords

Search

Navigation