Proposing a new model to aggregate ratings in multi-source feedback approach based on the evidence theory

  • Hossein Nahid Titkanloo
  • Abbas KeramatiEmail author
  • Roxana Fekri
Methodologies and Application


Researchers and practitioners in multi-source feedback (MSF) context generally use the average-based methods to aggregate ratings. Because of the uncertainties in the raters’ opinions, it is believed that the use of conventional averaging methods is not appropriate for aggregating MSF data. So, in MSF approach, there is a need to design a proper aggregation method that is capable to cope with the uncertainty in ratings. In this regard, in this paper, each rating group has been considered as a source of evidence, and a new aggregation model based on evidence theory has been proposed. In the proposed model, the collected data from each rating group by designing three different methods have been converted to the basic belief assignments and then aggregated using the Dempster rule of combination. In order to resolve the conflict between evidences, the discounting and compromise methods were used, and the output of the combination process was extracted using three different methods including the pignistic probability criterion, the plausibility transformation method and the expected value method. Finally, through a simulation study, the performance of the proposed model under various configurations was investigated. The results of the simulation study show that the proposed model, in almost all configurations, provides more accurate results than traditional aggregation method in MSF approach.


Multi-source feedback Uncertainty Dempster–Shafer theory Evidence theory 


Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. Anisseh M, Dodangeh J, Dashti MA, Piri F (2007) 360 degree personnel performance appraisal using the MADM models and presenting a model for overall ranking. In: IEEE international conference on industrial engineering and engineering managementGoogle Scholar
  2. Arbaiy N, Suradi Z (2007) Staff performance appraisal using fuzzy evaluation. In: Boukis C, Pnevmatikakis A, Polymenakos L (eds) Artificial intelligence and innovations 2007: from theory to applications. AIAI 2007. IFIP The International Federation for Information Processing, vol 247. Springer, Boston, MAGoogle Scholar
  3. Avazpour R, Ebrahimi E, Fathi MR (2013) A 360 degree feedback model for performance appraisal based on fuzzy AHP and TOPSIS. Int J Econ Manag Soc Sci 2(11):969–976Google Scholar
  4. Bostrom H, Johansson R, Karlsson A (2008) On evidential combination rules for ensemble classifiers. In: 2008 11th international conference on information fusion, pp 1–8Google Scholar
  5. Brutus SS (2010) Words versus numbers: a theoretical exploration of giving and receiving narrative comments in performance appraisal. Hum Resour Manag Rev 20(2):144–157CrossRefGoogle Scholar
  6. Chang J-R, Cheng C-H, Chen L-S (2007) A fuzzy-based military officer performance appraisal system. Appl Soft Comput 7(3):936–945CrossRefGoogle Scholar
  7. Cobb BR, Shenoy PP (2006) On the plausibility transformation method for translating belief function models to probability models. Int J Approx Reason 41(3):314–330MathSciNetzbMATHCrossRefGoogle Scholar
  8. Crispim-Junior CF, Ma Q, Fosty B, Romdhane R, Bremond F, Thonnat M (2015) Combining multiple sensors for event detection of older people. In: Health monitoring and personalized feedback using multimedia data. Springer, pp 179–194Google Scholar
  9. De Andrés R, García-Lapresta JL, González-Pachón J (2010) Performance appraisal based on distance function methods. Eur J Oper Res 207(3):1599–1607zbMATHCrossRefGoogle Scholar
  10. De Campos LM, Jorge M (1992) Characterization and comparison of Sugeno and Choquet integrals. Fuzzy Sets Syst 52(1):61–67MathSciNetzbMATHCrossRefGoogle Scholar
  11. De Sáa SDR, Gonz G, Teresa L (2015) Fuzzy rating scale-based questionnaires and their statistical analysis. IEEE Trans Fuzzy Syst 23(1):1–14CrossRefGoogle Scholar
  12. Dempster AP (1967) Upper and lower probabilities induced by a multi-valued mapping. Ann Math Stat 38(2):325–339MathSciNetzbMATHCrossRefGoogle Scholar
  13. Deng Y, Sadiq R, Jiang W, Tesfamariam S (2011) Risk analysis in a linguistic environment: a fuzzy evidential reasoning-based approach. Expert Syst Appl 38(12):15438–15446CrossRefGoogle Scholar
  14. Deng X, Li Y, Deng Y (2012) A decision making method based on Dempster–Shafer theory of evidence under the constraint of uncertain subjective information. J Inf Comput Sci 9(8):2049–2056MathSciNetGoogle Scholar
  15. Dezert J, Wang P, Tchamova A (2012) On the validity of Dempster–Shafer theory. In: 2012 15th international conference on information fusion (FUSION), pp 655–660Google Scholar
  16. Di Zio M, Vantaggi B (2017) Partial identification in statistical matching with misclassification. Int J Approx Reason 82(Suppl C):227–241MathSciNetzbMATHCrossRefGoogle Scholar
  17. Esen H, Hatipoğlu T, Boyacı Aİ (2016) A fuzzy approach for performance appraisal: the evaluation of a purchasing specialist. In: Merelo J, Rosa A, Cadenas J, Dourado A, Madani K, Filipe J (eds) Computational Intelligence. Studies in Computational Intelligence, vol 620. Springer, ChamGoogle Scholar
  18. Espinilla M, De Andrés R, Martínez FJ, Martínez L (2013) A 360-degree performance appraisal model dealing with heterogeneous information and dependent criteria. Inf Sci 222:459–471MathSciNetCrossRefGoogle Scholar
  19. Fassinut-Mombot B, Choquel J-B (2000) An entropy method for multisource data fusion. In: Proceedings of the third international conference on information fusion, 2000. FUSION 2000, vol 2, pp THC5–THC17Google Scholar
  20. Ferson S, Kreinovich V, Ginzburg L, Myers DS, Sentz K (2003) Constructing probability boxes and Dempster–Shafer structures, Sandia National Laboratories, Albuquerque, NM, SAND2002-4015Google Scholar
  21. Florea MC, Jousselme AL, Bossé É, Grenier D (2009) Robust combination rules for evidence theory. Inf Fusion 10(2):183–197CrossRefGoogle Scholar
  22. Golman R, Bhatia S (2012) Performance evaluation inflation and compression. Account Organ Soc 37(8):534–543CrossRefGoogle Scholar
  23. Ha-Duong M (2008) Hierarchical fusion of expert opinions in the Transferable Belief Model, application to climate sensitivity. Int J Approx Reason 49(3):555–574CrossRefGoogle Scholar
  24. Hassan S, Rohrbaugh J (2009) Incongruity in 360-degree feedback ratings and competing managerial values: evidence from a public agency setting. Int Public Manag J 12(4):421–449CrossRefGoogle Scholar
  25. Hooft EAJ, Flier H, Minne MR (2006) Construct validity of multi-source performance ratings: an examination of the relationship of self-, supervisor-, and peer-ratings with cognitive and personality measures. Int J Sel Assess 14(1):67–81CrossRefGoogle Scholar
  26. Jiang W, Wei B, Xie C, Zhou D (2016) An evidential sensor fusion method in fault diagnosis. Adv Mech Eng 8(3):1687814016641820Google Scholar
  27. Kahraman C, Öztayşi B (2013) Personnel selection using type-2 fuzzy ahp method. Bus Manag Rev 4(1):118–126Google Scholar
  28. Lefèvre E, Elouedi Z (2013) How to preserve the conflict as an alarm in the combination of belief functions? Decis Support Syst 56(1):326–333CrossRefGoogle Scholar
  29. Lefevre E, Colot O, Vannoorenberghe P (2002) Belief functions combination and conflict management. Inf Fusion J 3(2):149–162CrossRefGoogle Scholar
  30. Leung Y, Ji N-N, Ma J-H (2013) An integrated information fusion approach based on the theory of evidence and group decision-making. Inf Fusion 14(4):410–422CrossRefGoogle Scholar
  31. Levy PE, Williams JR (2004) The social context of performance appraisal: a review and framework for the future. J Manag 30(6):881–905Google Scholar
  32. Liu W (2006) Analyzing the degree of conflict among belief functions. Artif Intell 170(11):909–924MathSciNetzbMATHCrossRefGoogle Scholar
  33. Liu ZG, Dezert J, Pan Q, Mercier G (2011) Combination of sources of evidence with different discounting factors based on a new dissimilarity measure. Decis Support Syst 52(1):133–141CrossRefGoogle Scholar
  34. Lubiano MA, De La Rosa De S, Sáa M, Montenegro B Sinova, Gil MÁ (2016) Descriptive analysis of responses to items in questionnaires. Why not using a fuzzy rating scale? Inf. Sci. (Ny) 360:131–148CrossRefGoogle Scholar
  35. Ma Y, Chandler JS, Wilkins DC et al (1991) On the decision making problem in Dempster–Shafer theoryGoogle Scholar
  36. Markham SE, Smith JW, Markham IS, Braekkan KF, Witt J, Markham IS, Braekkan KF (2014) A new approach to analyzing the Achilles’ heel of multisource feedback programs: can we really trust ratings of leaders at the group level of analysis? Leadersh Q 25(6):1120–1142CrossRefGoogle Scholar
  37. Min-peng X, Xiao-hu Z, Xin D (2012) Modeling of engineering R&D staff performance appraisal model based on fuzzy comprehensive evaluation. Syst Eng Procedia 4:236–242CrossRefGoogle Scholar
  38. Mohammadi SE, Makui A (2017) Multi-attribute group decision making approach based on interval-valued intuitionistic fuzzy sets and evidential reasoning methodology. Soft Comput 21(17):5061–5080zbMATHCrossRefGoogle Scholar
  39. Moon C, Lee J, Lim S (2010) A performance appraisal and promotion ranking system based on fuzzy logic: an implementation case in military organizations. Appl Soft Comput 10(2):512–519CrossRefGoogle Scholar
  40. Nowack KM, Mashihi S (2012) Evidence-based answers to 15 questions about leveraging 360-degree feedback. Consult Psychol J Pract Res 64(3):157CrossRefGoogle Scholar
  41. Oukhellou L, Debiolles A, Denœux T, Aknin P (2010) Fault diagnosis in railway track circuits using Dempster–Shafer classifier fusion. Eng Appl Artif Intell 23(1):117–128CrossRefGoogle Scholar
  42. Ozkan C, Keskin GA, Omurca SI (2014) A variant perspective to performance appraisal system: fuzzy c-means algorithm. Int J Ind Eng Theor Appl Pract 21(3):168–178Google Scholar
  43. Raufaste E, Da Silva Neves R, Mariné C (2003) Testing the descriptive validity of possibility theory in human judgments of uncertainty. Artif Intell 148(1):197–218zbMATHCrossRefGoogle Scholar
  44. Rogova GL, Nimier V (2004) Reliability in information fusion: literature survey. In: Proceedings of seventh international conference information fusion, FUSION 2004, vol 2, pp 1158–1165Google Scholar
  45. Sentz K, Ferson S (2002) Combination of evidence in Dempster–Shafer theory, vol 4015. CiteseerGoogle Scholar
  46. Sepehrirad R, Azar A, Sadeghi A (2012) Developing a hybrid mathematical model for 360-degree performance appraisal: a case study. Procedia Soc Behav Sci 62:844–848CrossRefGoogle Scholar
  47. Shafer G (1976) A mathematical theory of evidence, vol 1. Princeton University Press, PrincetonzbMATHGoogle Scholar
  48. Smarandache F (2004) An in-depth look at information fusion rules & the unification of fusion theories. arXiv: Cs/0410033Google Scholar
  49. Smets P (2002) Decision making in a context where uncertainty is represented by belief functions. Belief Funct Bus Decis 17:1–47zbMATHGoogle Scholar
  50. Smets P (2005) Decision making in the TBM: the necessity of the pignistic transformation. Int J Approx Reason 38(2):133–147MathSciNetzbMATHCrossRefGoogle Scholar
  51. Smets P (2007) Analyzing the combination of conflicting belief functions. Inf Fusion 8(4):387–412CrossRefGoogle Scholar
  52. Stahl D-IJ (1996) A method for personnel selection in concurrent engineering using fuzzy sets. In: Sebastian HJ, Antonsson EK (eds) Fuzzy sets in engineering design and configuration. Springer, Berlin, pp 265–276CrossRefGoogle Scholar
  53. Strat TM (1990) Decision analysis using belief functions. Int J Approx Reason 4(5–6):391–417zbMATHCrossRefGoogle Scholar
  54. Tang D, Wong TC, Chin KS, Kwong CK (2014) Evaluation of user satisfaction using evidential reasoning-based methodology. Neurocomputing 142:86–94CrossRefGoogle Scholar
  55. Taroun A, Yang J-B (2011) Dempster–Shafer theory of evidence: potential usage for decision making and risk analysis in construction project management. Built Hum Environ Rev 4:155–166Google Scholar
  56. Tosti DT, Addison RM (2009) 360-degree feedback: going around in circles? Perform Improv 48(3):36–39CrossRefGoogle Scholar
  57. Vasu JZ, Deb AK, Mukhopadhyay S (2015) MVEM-based fault diagnosis of automotive engines using Dempster–Shafer theory and multiple hypotheses testing. IEEE Trans Syst Man Cybern Syst 45(7):977–989CrossRefGoogle Scholar
  58. Xie J, Zeng W, Li J, Yin Q (2017) Similarity measures of generalized trapezoidal fuzzy numbers for fault diagnosis. Soft Comput 23:1999–2014zbMATHCrossRefGoogle Scholar
  59. Xu D-L (2012) An introduction and survey of the evidential reasoning approach for multiple criteria decision analysis. Ann Oper Res 195(1):163–187MathSciNetzbMATHCrossRefGoogle Scholar
  60. Xu H-Y, Yue Z-H, Wang C, Dong K, Pang H-S, Han Z (2017) Multi-source data fusion study in scientometrics. Scientometrics 111(2):773–792CrossRefGoogle Scholar
  61. Yager RR, Alajlan N (2015) Dempster–Shafer belief structures for decision making under uncertainty. Knowl Based Syst 80:58–66CrossRefGoogle Scholar
  62. Yang J-B, Xu D-L (2002) On the evidential reasoning algorithm for multiple attribute decision analysis under uncertainty. IEEE Trans Syst Man Cybern Part A Syst Hum 32(3):289–304CrossRefGoogle Scholar
  63. Yanushkevich SN, Gavrilova ML, Shmerko VP, Lyshevski SE, Stoica A, Yager RR (2011) Belief trees and networks for biometric applications. Soft Comput 15(1):3–11CrossRefGoogle Scholar
  64. Yue S, Wang J, Li B, Wang H (2009) Using Dempster–Shafer evidence theory and choquet integral for image segmentation in gas-liquid two-phase flow. Int J Distrib Sens Netw 5(1):53CrossRefGoogle Scholar
  65. Zadeh LA (1986) A simple view of the Dempster–Shafer theory of evidence and its implication for the rule of combination. AI Mag. 7(2):85Google Scholar
  66. Zhu H, Basir O (2006) A novel fuzzy evidential reasoning paradigm for data fusion with applications in image processing. Soft Comput 10(12):1169–1180zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Industrial EngineeringPayame Noor UniversityTehranIran
  2. 2.Ted Rogers School of ManagementRyerson UniversityTorontoCanada
  3. 3.Industrial EngineeringUniversity of TehranTehranIran

Personalised recommendations