Abstract
Diagrams are an essential mode of thinking and communication. From the proverbial napkin sketch to office whiteboards to formal CAD drawings, we think and collaborate around sketches.
My group and I at MIT have been working for a decade to enable computers to join the conversation. In this talk I explore what it has taken to get computers to understand sketches and consider what it means for them to understand. In reviewing systems we have built and domains we have explored, I describe several themes that have emerged from our work. First, we have found that natural interaction is knowledge- based. Our ease of interacting with one another, for example, arises in large measure from a substantial body of shared knowledge. What is manifestly true of people appears to be true for computers as well. Second, there is considerable power in using multiple representations. Viewing sketches from the perspective of their spatial, temporal, and conceptual representations, and the interconnections among those representations, offers an effective means of dealing with challenges like noise and ambiguity. Third, diagrams are not enough. Think of all the whiteboard conversations you’ve had, and consider what got written on the whiteboard. Never mind that things get erased; even with multiple snapshots or a continuous record, what could you understand if all you had afterward was the visual record? This leads to the last theme: putting diagrams in the larger content of human communication as it routinely happens, i.e., communication that is multimodal, conversational, symmetric, and mixed initiative. I explore all of these issues and illustrate them by describing a variety of systems we have constructed.
Chapter PDF
Similar content being viewed by others
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Davis, R. (2010). Understanding Diagrams, and More: The Computer’s View. In: Goel, A.K., Jamnik, M., Narayanan, N.H. (eds) Diagrammatic Representation and Inference. Diagrams 2010. Lecture Notes in Computer Science(), vol 6170. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-14600-8_2
Download citation
DOI: https://doi.org/10.1007/978-3-642-14600-8_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-14599-5
Online ISBN: 978-3-642-14600-8
eBook Packages: Computer ScienceComputer Science (R0)