Beyond the Desktop Metaphor: Toward More Effective Display, Interaction, and Telecollaboration in the Office of the Future via a Multitude of Sensors and Displays

Abstract

We are engaged in a long-term project to improve personal productivity for computer-related activities and tele-collaboration in an office environment of the future. Personal computer-related activities, we believe will be enhanced by capability to project imagery on any surface in the office, that together with precise head and eye-tracking, will enable head-tracked stereo imagery to be added to the user’s views of his/her office environment — creating a 3D immersive generalization of the now ubiquitous 2D desktop metaphor as the principal human-computer interface. We plan to realize this kind of system by mounting many video projectors and video cameras around the room, especially around the ceiling. The projectors may provide the only source of light in the room and will allow detailed imagery to be projected (almost) everywhere in the office. In order to generate the appropriate imagery, however, a detailed 3D map of the changing office environment needs to be acquired. This will be acquired by measuring, with synchronized cameras and projectors, the precise 3D location(s) of the surface(s) light up by each pixel of each projector. Local collaboration will be enhanced by tracking each of several individuals in the office and generating (by time-division multiplexing or by other means), a stereo image pair appropriate for each individual. Objects under design may be displayed, for each individual, from his/her own perspective and to his/her own specifications of interest. TELE-collaboration activities, we believe will be enhanced by having such an enhanced office environment for each of the small group of distant collaborators, and displaying for each participant, in addition to the shared objects under discussion, some combination of the remote scenes that include the changing 3D images of each of the participants and 3D images of physical objects of joint interest. To realize many of these capabilities, each user may need to wear polarized eyeglasses to perceive proper stereo imagery. Although initial results are encouraging, numerous difficult problems remain — how, for example, can imagery be projected onto a dark-colored surface in the room. The cost of such systems, with many projectors and cameras, image generators and image acquisition devices, may initially be prohibitively expensive, but is expected to decrease as the cost of such off-the-shelf equipment naturally decreases with increased market size. The positive psychological effects of working and interacting in such an immersive environment within a “standard” office will be so compelling, we believe, that users will not readily wish to return to working within the constraints of a 21” monitor. Much of this work is being carried out as part of a collaboration among the five sites of the NSF Science and Technology Center in Computer Graphics and Scientific Visualization (Brown, Caltech, Cornell, UNC, and Utah) and is also being carried on in collaboration with the GRASP Lab at University of Pennsylvania, and is part of the National Teleimmersion Initiative sponsored by Advanced Network and Services.