A Series of Experiments into the Use of Pairwise Comparison Techniques to Evaluate Criteria Weights

Abstract

This paper describes a series of experiments to investigate the nature of responses to questions to elicit the relative importance of criteria in a multi-attribute evaluation. The experiments were designed to investigate the behaviour of decision makers under various questioning procedures. The subjects were students in knitwear design and in engeering at Trent Polytechnic. We feel that the pairwise comparison technique used in Saaty’s AHP is a powerful means of eliciting such judgments and the experiments were carried out within that framework.

However, there is considerable concern that the questions asked to elicit such judgments are often meaningless or have an implicit meaning of which the decison maker is unaware and/or which he may not be able to understand. Nevertheless, decision makers are often willing to give answers to such questions and furthermore have confidence in those answers. We believe that the concept of ‘weights’ should be related to the scales on which objects are evaluated and that it is the objects themselves which define those scales. Thus, a change in the set of objects to be evaluated, by the removal of addtion of one or more objects may, in theory, cause a corresponding change in the weights assigned to the criteria. The nature of the change is dependent upon the decision makers’ interpretation of ‘weight’.