Abstract
We compare four common data collection techniques to elicit preferences: the rating of items, the ranking of items, the partitioning of a given amount of points among items, and a reduced form of the technique for comparing items in pairs. University students were randomly assigned a questionnaire employing one of the four techniques. All questionnaires incorporated the same collection of items. The data collected with the four techniques were converted into analogous preference matrices, and analyzed with the Bradley–Terry model. The techniques were evaluated with respect to the fit to the model, the precision and reliability of the item estimates, and the consistency among the produced item sequences. The rating, ranking and budget partitioning techniques performed similarly, whereas the reduced pair comparisons technique performed a little worse. The item sequence produced by the rating technique was very close to the sequence obtained averaging over the three other techniques.
Similar content being viewed by others
References
Agresti, A.: An Introduction to Categorical Data Analysis, 2nd edn. Wiley, Hoboken (2007)
Aloysius, J.A., Fred, D.D., Darryl, D.W., Taylor, A.R., Kottemann, J.E.: User acceptance of multi-criteria decision support systems: the impact of preference elicitation techniques. Eur. J. Oper. Res. 169, 273–285 (2006)
Alwin, D.F., Krosnick, J.A.: The measurement of values in surveys: a comparison of ratings and rankings. Public Opin. Q. 49, 535–552 (1985)
Andrich, D.: A rating formulation for ordered response categories. Psychometrika 43, 561–573 (1978)
Bech, M., Gyrd-Hansen, D., Kjær, T., Lauridsen, J.T., Sørensen, J.: Graded pairs comparison—does strength of preference matter? Analysis of preferences for specialised nurse home visits for pain management. Health Econ. 16, 513–529 (2007)
Bollen, K.A.: Structural Equations with Latent Variables. Wiley, New York (1989)
Bradburn, N.M., Sudman, S., Wansink, B.: Asking Questions: The Definitive Guide to Questionnaire Design—For Market Research, Political Polls, and Social and Health Questionnairs, Revised edn. Jossey Bass, San Francisco (2004)
Bradley, R.A., Terry, M.E.: Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika 39, 324–345 (1952)
Coombs, C.H.: A Theory of Data. Wiley, Oxford (1964)
David, H.A.: The Method of Paired Comparisons, 2nd edn. Chapman and Hall, London (1988)
Elrod, T., Louviere, J.J., Davey, K.S.: An empirical comparison of ratings-based and choice-based conjoint models. J. Mark. Res. 29, 368–377 (1992)
Fabbris, L.: Measurement scales for scoring or ranking sets of interrelated items. In: Davino, C., Fabbris, L. (eds.) Survey Data Collection and Integration, pp. 21–44. Springer, Heidelberg (2013)
Feather, N.T.: The measurement of values: effects of different assessment procedures. Aust. J. Psychol. 25, 221–231 (1973)
Fienberg, S.E., Larntz, K.: Log linear representation for paired and multiple comparisons models. Biometrika 63, 245–254 (1976)
Fisher Jr., W.P.: Reliability, separation, strata statistics. Rasch Meas. Trans. 6, 238 (1992)
Guttman, L.: An approach for quantifying paired-comparisons and rank order. Ann. Math. Stat. 17, 143–163 (1946)
Hauser, J.R., Rao, V.: Conjoint analysis, related modeling, and applications. In: Wind, Y., Green, P.E. (eds.) Marketing Research and Modeling: Progress and Prospects: A Tribute to Paul E. Green, pp. 141–158. Springer, New York (2004)
Huber, P.J.: Pairwise comparison and ranking: optimum properties of the row sum procedure. Ann. Math. Stat. 34, 511–520 (1963)
Huber, J., Wittink, D.R., Fiedler, J.A., Miller, R.: The effectiveness of alternative preference elicitation procedures in predicting choice. J. Mark. Res. 30, 105–114 (1993)
Jech, T.: A quantitative theory of preferences: some results on transition functions. Soc. Choice Welf 6, 301–314 (1989)
Krosnick, J.A., Alwin, D.F.: A test of the form-resistant correlation hypothesis: ratings, rankings, and the measurement of values. Public Opin. Q. 52, 526–538 (1988)
Linacre, J.M.: What do infit and outfit, mean-square and standardized mean? Rasch Meas. Trans. 16, 878 (2002)
Linacre, J.M.: Facets Computer Program for Many-Facet Rasch Measurement, Version 3.70.0. Winsteps.com, Beaverton (2012)
Louviere, J.J., Hensher, D.A., Swait, J.D.: Stated Choice Methods. Analysis and Application. Cambridge University Press, Cambridge (2003)
Luce, R.D.: Individual Choice Behavior: A Theoretical Analysis. Wiley, New York (1959)
Maio, G.R., Roese, N.J., Seligman, C., Katz, A.: Rankings, ratings, and the measurement of values: evidence for the superior validity of ratings. Basic Appl. Soc. Psychol. 18, 171–181 (1996)
McFadden, D.: The choice theory approach to market research. Mark. Sci. 5, 275–297 (1986)
Smith Jr., E.V.: Evidence for the reliability of measures and validity of measure interpretation: a Rasch measurement perspective. J. Appl. Meas. 2, 281–311 (2001)
Takane, Y.: Maximum likelihood additivity analysis. Psychometrika 17, 225–240 (1982)
Takane, Y.: Analysis of covariance structures and probabilistic binary choice data. In: de Soete, G., Feger, H., Klauer, K.C. (eds.) New Developments in Psychological Choice Modeling, pp. 139–160. North Holland, Amsterdam (1989)
Thurstone, L.L.: A law of comparative judgment. Psychol. Rev. 34, 281–299 (1927)
Torgerson, W.S.: Theory and Methods of Scaling. Wiley, New York (1958)
Tversky, A., Russo, J.E.: Substitutability and similarity in binary choices. J. Math. Psychol. 6, 1–12 (1969)
Author information
Authors and Affiliations
Corresponding author
Appendix: List of the university services submitted to evaluation
Appendix: List of the university services submitted to evaluation
-
A.
Counselling high school graduates for university choices, links with high school
-
B.
Counselling for course changing (during courses)
-
C.
Toward labour guidance (after graduation)
-
D.
Organizational support of studies (reducing costs, improving canteen and accommodation, etc.)
-
E.
Amusement, socialization, culture, sports and other types of relationships with guest town
-
F.
Information exchange for and among students (call centre, internet, help desks, etc.)
-
G.
Economic support for deserving students
-
H.
Larger classrooms and (possibly self-managed) rooms for besides-study activities
-
I.
Teaching materials (lecture notes, online textbooks, library accessibility, etc.) to improve study efficacy
-
J.
Learning supporting activities (internships, study groups, summer schools, language courses)
-
K.
Individual or group tutorship to improve learning
-
L.
Increasing opportunities to study or work abroad (Erasmus, Leonardo, etc.)
Rights and permissions
About this article
Cite this article
Anselmi, P., Fabbris, L., Martini, M.C. et al. Comparison of four common data collection techniques to elicit preferences. Qual Quant 52, 1227–1239 (2018). https://doi.org/10.1007/s11135-017-0514-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-017-0514-7