An Empirical Analysis of Assessment Errors for Weights and Andness in LSP Criteria

  • Jozo J. Dujmović
  • Wen Yuan Fang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3131)


We investigate the accuracy of parameters in the Logic Scoring of Preference (LSP) criterion functions for system evaluation. Main parameters are weights and conjunction/disjunction degrees (andness/orness). Weights reflect the level of relative importance of various decision variables. Andness/orness describes a desired level of simultaneity/replaceability in satisfying component criteria. These parameters are assessed by one or more professional evaluators and their values differ from (usually unknown) optimum values. In this paper we identify all potential sources of errors in LSP criterion functions. Our goal is to investigate the distribution of errors, their average values, and the quality of individual evaluators and their teams.


Elementary Criterion Micro Model Average Absolute Error Global Preference Assessment Error 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barron, F.H., Barrett, B.E.: The efficacy of SMART – Simple Multi- Attribute Rating Technique Extended to Ranking. Acta Psychologica 93, 23–36 (1996)CrossRefGoogle Scholar
  2. 2.
    Belton, V., Stewart, T.J.: Multiple Criteria Decision Analysis: an Integrated Approach. Kluwer Academic Publishers, Dordrecht (2002)Google Scholar
  3. 3.
    Bottomley, P.A., Doyle, J.R.: A Comparison of Three Weight Elicitation Methods: Good, Better, and Best. Omega 29, 553–560 (2001)CrossRefGoogle Scholar
  4. 4.
    Butler, J., Jia, J., Dyer, J.: Simulation Techniques for the Sensitivity Analysis of Multi- Criteria Decision Models. European Journal of Operational Research 103, 531–546 (1997)zbMATHCrossRefGoogle Scholar
  5. 5.
    Dujmović, J.J.: Correlational Aspects of Error Compensation in the Weighted Scoring Method for Selection of Data Processing Systems (In Serbo-Croatian). In: Proceedings of the 7th Informatica Conference (1972)Google Scholar
  6. 6.
    Dujmović, J.J.: A Generalization of Some Functions in Continuous Mathematical Logic (In Serbo-Croatian) .In: Proceedings of the Informatica Conference, Bled, Yugoslavia (1973)Google Scholar
  7. 7.
    Dujmović, J.J.: Mixed Averaging by Levels (MAL)–A System and Computer Evaluation Method (In Serbo-Croatian) .In: Proceedings of the Informatica Conference, Bled, Yugoslavia (1973)Google Scholar
  8. 8.
    Dujmović, J.: Weighted Conjunctive and Disjunctive Means and their Application in System Evaluation. Journal of the University of Belgrade, EE Dept., Series Mathematics and Physics 483, 147–158 (1974)Google Scholar
  9. 9.
    Dujmović, J.J.: Extended Continuous Logic and the Theory of Complex Criteria. Journal of the University of Belgrade, EE Dept., Series Mathematics and Physics 537, 197–216 (1975)Google Scholar
  10. 10.
    Dujmović, J.J.: Partial Absorption Function. Journal of the University of Belgrade, EE Dept., Series Mathematics and Physics 659, 156–163 (1980)Google Scholar
  11. 11.
    Dujmović, J.J., Elnicki, R.: A DMS Cost/Benefit Decision Model: Mathematical Models for Data Management System Evaluation, Comparison, and Selection. National Bureau of Standards, Washington D.C., No. GCR 82-374. NTIS No. PB 82-170150, p.150 (1982)Google Scholar
  12. 12.
    Dujmović, J.J.: Preferential Neural. In: Antognetti, P., Milutinović, V. (eds.) Neural Networks - Concepts, Applications, and Implementations. Prentice-Hall Advanced Reference Series, vol. . II,ch.7, pp. 155–206. Prentice-Hall, Englewood Cliffs (1991)Google Scholar
  13. 13.
    Filev, D.P., Yager, R.R.: On the Issue of Obtaining OWA Operator Weights. Fuzzy Sets and Systems 94, 157–169 (1998)CrossRefMathSciNetGoogle Scholar
  14. 14.
    Gilb, T.: Software Metrics. Winthrop Publishers (1977)Google Scholar
  15. 15.
    Jessop, A.: Sensitivity and Robustness in Selection Problems. Computers & Operations Research 31, 607–622 (2004)zbMATHCrossRefMathSciNetGoogle Scholar
  16. 16.
    Martin, R.A., Morrison, S.A.: A Software Quality Assessment Methodology for Evaluating System Lifecycle Risks. MITRE Corporation (January 1994)Google Scholar
  17. 17.
    Miller, G.A.: The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. The Psychological Review 63, 81–97 (1956)CrossRefGoogle Scholar
  18. 18.
    Roberts, R., Goodwin, P.: Weight Approximations in Multi-attribute Decision Models. Journal of Multi-Criteria Decision Analysis 11, 291–303 (2002)zbMATHCrossRefGoogle Scholar
  19. 19.
    Rouder, J.N., Morey, R.D., Cowan, N., Pfalz, M.: Learning in Unidimensional Absolute Identification (2004),
  20. 20.
    Scharf(Gilb), T.: Weighted Ranking by Levels. IAG Journal 2(2), 7–18 (1969)Google Scholar
  21. 21.
    Scharf (Gilb), T.: Weighted Ranking by Levels Computer Evaluation Method – One Year of Experience. IAG Journal 3(3), 71–91 (1970)Google Scholar
  22. 22.
    Stewart, T.J.: Robustness of Additive Value Function Methods in MCDM. Journal of Multi-Criteria Decision Analysis 5, 301–309 (1996)zbMATHCrossRefGoogle Scholar
  23. 23.
    Su, S.Y.W., Dujmovič, J.J., Batory, D.S., Navathe, S.B., Elnicki, R.: A Cost-Benefit Decision Model: Analysis, Comparison, and Selection of Data Management Systems. ACM Transactions on Database Systems 12(3), 472–520 (1987)CrossRefGoogle Scholar
  24. 24.
    Torra, V.: Learning Weights for the Quasi-Weighted Mean. IEEE Trans. on Fuzzy Systems 10(5), 653–666 (2002)CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Jozo J. Dujmović
    • 1
  • Wen Yuan Fang
    • 1
  1. 1.Department of Computer ScienceSan Francisco State UniversitySan FranciscoUSA

Personalised recommendations