A Syntax-Directed Approach to Picture Semantics

Abstract

If one constructs a visual language, it is necessary to have a methodology for constructing the relationship between aspects of the pictures and the corresponding aspects of the application domain. The various methods for specifying visual languages emphasize computational aspects, where the semantics of the picture is embedded in the computational setting of the specification, but they are not based on an understanding of the cognitive issues involved in the semantics of pictures and in the use of the pictures for a better grasp of the application domain and for manipulating the domain. In our opinion, it is the analogy between a picture and its meaning that can help people to understand the meaning represented by the picture and that the match between the syntactic structures of the picture and what it represents is an important part of the analogy. We formalise the notion of matching in an approach to picture semantics based on order-sorted algebra. Pictures are described in a well-structured framework (order sorted signature) and so is the application domain. The relationship between pictures and their meanings is formalised by the notion of a signature morphism which, together with the formal description of pictures, enforces a structural match between pictures and what they represent. We also discuss various issues brought out by this algebraic approach and try to assess the consequences for visual languages.