Advertisement

Electronic Commerce Research

, Volume 8, Issue 1–2, pp 1–27 | Cite as

Evaluating product search and recommender systems for E-commerce environments

  • Pearl Pu
  • Li Chen
  • Pratyush Kumar
Article

Abstract

Online systems that help users select the most preferential item from a large electronic catalog are known as product search and recommender systems. Evaluation of various proposed technologies is essential for further development in this area. This paper describes the design and implementation of two user studies in which a particular product search tool, known as example critiquing, was evaluated against a chosen baseline model. The results confirm that example critiquing significantly reduces users’ task time and error rate while increasing decision accuracy. Additionally, the results of the second user study show that a particular implementation of example critiquing also made users more confident about their choices. The main contribution is that through these two user studies, an evaluation framework of three criteria was successfully identified, which can be used for evaluating general product search and recommender systems in E-commerce environments. These two experiments and the actual procedures also shed light on some of the most important issues which need to be considered for evaluating such tools, such as the preparation of materials for evaluation, user task design, the context of evaluation, the criteria, the measures and the methodology of result analyses.

Keywords

Preference-based search Product recommender systems Example critiquing interfaces Decision technology Electronic product catalog Tradeoff analysis Fisheye view interfaces 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bederson, B. B. (2000). Fisheye Menus. In Proceedings of ACM conference on user interface software and technology (UIST 2000) (pp. 217–226). New York: Assoc. Comput. Mach. CrossRefGoogle Scholar
  2. 2.
    Bederson, B. B., et al. (2004). DateLens: a fisheye calendar interface for PDAs. ACM Transactions on Computer-Human Interaction, 11(1), 90–119. CrossRefGoogle Scholar
  3. 3.
    Burke, R. (2002). Hybrid recommender systems: survey and experiments. User Modeling and User-Adapted Interaction, 12(4), 331–370. CrossRefGoogle Scholar
  4. 4.
    Burke, R., Hammond, K., & Young, B. (1997). The findme approach to assisted browsing. IEEE Expert: Intelligent Systems and Their Applications, 12(4), 32–40. Google Scholar
  5. 5.
    Callahan, E., & Koenemann, J. (2000). A comparative usability evaluation of user interfaces for online product catalogs. In Proceedings of ACM electronic commerce conference (pp. 197–206). New York: Assoc. Comput. Mach. CrossRefGoogle Scholar
  6. 6.
    Carenini, G., & Poole, D. (2002). Constructed preferences and value-focused thinking: implications for AI research on preference elicitation. In AAAI-02 workshop on preferences in AI and CP: symbolic approaches, Edmonton, Canada. Google Scholar
  7. 7.
    Faltings, B., Torrens, M., & Pu, P. (2004). Solution generation with qualitative models of preferences. International Journal of Computational Intelligence and Applications, 20(2), 246–263. Google Scholar
  8. 8.
    Fischer, G. W., & Hawkins, S. A. (1993). Strategy compatibility, scale compatibility, and the prominence effect. Journal of Experimental Psychology: Human Perception and Performance, 19, 580–597. CrossRefGoogle Scholar
  9. 9.
    Furnas, G. W. (1986). Generalized fisheye views. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 16–23). New York: Assoc. Comput. Mach. Google Scholar
  10. 10.
    Janecek, P., & Pu, P. (2002). A framework for designing fisheye views to support multiple semantic contexts. In Proceedings of the international conference on advanced visual interfaces (AVI’02). New York: Assoc. Comput. Mach. Google Scholar
  11. 11.
    Jedetski, J., Adelman, L., & Yeo, C. (2002). How web site decision technology affects consumers. Internet Computing, 6(2), 72–79. CrossRefGoogle Scholar
  12. 12.
    Ha, V., & Haddawy, P. (2003). Similarity of personal preferences: theoretical foundations and empirical analysis. Artificial Intelligence, 46(2), 9–173. Google Scholar
  13. 13.
    Haubl, G., & Trifts, V. (2000). Consumer decision making in online shopping environments: the effects of interactive decision aids. Marketing Science, 19(1), 4–21. CrossRefGoogle Scholar
  14. 14.
    Herlocker, J., Konstan, J., Terveen, L. G., & Riedl, J. T. (2004). Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems, 22(1), 5–53. CrossRefGoogle Scholar
  15. 15.
    Hogarth, R. (1987). Judgment and choice. New York: Wiley. Google Scholar
  16. 16.
    Keeney, R. L., & Raiffa, H. (1976). Decisions with multiple objectives: preferences and value tradeoffs. New York: Wiley. Google Scholar
  17. 17.
    Lamping, J., Rao, R., & Pirolli, P. (1995). A focus + context technique based on hyperbolic geometry for visualizing large hierarchies. In CHI’95, ACM conference on human factors in computing systems (pp. 401–408). Google Scholar
  18. 18.
    Linden, G., Hanks, S., & Lesh, N. (1997). Interactive assessment of user preference models: the automated travel assistant. In Proceedings of user modeling’97 (pp. 67–78). Google Scholar
  19. 19.
    Maes, P., Guttman, R., & Moukas, A. (1999). Agents that buy and sell: transforming commerce as we know it. Communications of the ACM, 42(3), 81–91. CrossRefGoogle Scholar
  20. 20.
    McCarthy, K. et al. (2004). On the dynamic generation of compound critiques in conversational recommender systems. In Proceedings of the third international conference on adaptive hypermedia and adaptive web-based systems. Google Scholar
  21. 21.
    McCarthy, K., et al. (2005). Experiments in dynamic critiquing. In Proceedings of the international conference on intelligent user interfaces (IUI05) (pp. 175–182). Google Scholar
  22. 22.
    Payne, J. W., Bettman, J. R., & Johnson, E. J. (1993). The adaptive decision maker. Cambridge: Cambridge University Press. Google Scholar
  23. 23.
    Pirolli, P., Card, S. K., & Wege, M. M. (2003). The effects of information scent on visual search in the hyperbolic tree browser. ACM Transactions on Computer-Human Interaction, 10(1), 20–53. CrossRefGoogle Scholar
  24. 24.
    Pu, P., & Faltings, B. (2000). Enriching buyers’ experiences: the SmartClient approach. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 289–296). New York: Assoc. Comput. Mach. CrossRefGoogle Scholar
  25. 25.
    Pu, P., & Faltings, B. (2004). Interface technologies for decision tradeoff problems using CSP. International Journal of Constraints, 9(4), 289–310. CrossRefGoogle Scholar
  26. 26.
    Pu, P., & Kumar, P. (2004). Evaluating example-based search tools. In Proceedings of the ACM conference on electronic commerce (EC’04) (pp. 208–217). New York: Assoc. Comput. Mach. CrossRefGoogle Scholar
  27. 27.
    Rao, R., & Card, S. K. (1994). The table lens: merging graphical and symbolic representations in an interactive focus + context visualization for tabular information. In Proceeding of Human factors in computing systems (CHI’94) (pp. 318–322). New York: Assoc. Comput. Mach. Google Scholar
  28. 28.
    Reilly, J., et al. (2004). Dynamic critiquing. In Proceedings of the seventh European conference on case-based reasoning (ECCBR-04). Google Scholar
  29. 29.
    Reilly, J., et al. (2004). Incremental critiquing. In Proceedings of the 24th SGAI international conference on innovative techniques and applications of artificial intelligence (AI-04). Google Scholar
  30. 30.
    Resnick, P. et al. (1994). Grouplens: an open architecture for collaborative filtering of netnews. In Proceedings of the ACM conference on computer supported cooperative work (pp. 130–137). New York: Assoc. Comput. Mach. Google Scholar
  31. 31.
    Sarkar, M., & Brown, M. H. (1992). Graphical fisheye views of graphs. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 83–91). New York: Assoc. Comput. Mach. Google Scholar
  32. 32.
    Smyth, B., et al. (2004). Compound critiques feedback for conversational recommender systems. In Proceedings of the IEEE/WIC/ACM international conference on web intelligence (WI-04). Google Scholar
  33. 33.
    Spence, R., & Apperly, M. (1982). Data base navigation: an office environment for the professional. Behaviour and Information Technology, 1(1), 43–54. CrossRefGoogle Scholar
  34. 34.
    Spiekermann, S., & Paraschiv, C. (2002). Motivating human-agent interaction: transferring insights from behavioral marketing to interface design. Journal of Electronic Commerce Research, 2, 255–285. CrossRefGoogle Scholar
  35. 35.
    Stolze, M. (2000). Soft navigation in electronic product catalogs. International Journal on Digital Libraries, 3(1), 60–66. CrossRefGoogle Scholar
  36. 36.
    Torrens, M., & Faltings, B. (1999). SmartClients: constraint satisfaction as a paradigm for scaleable intelligent information systems. In Workshop notes, artificial intelligence for electronic commerce, the sixteenth national conference on artificial intelligence (AAAI’99) (pp. 10–15). Merlo Park: AAAI Press. Google Scholar
  37. 37.
    Torrens, M., Faltings, B., & Pu, P. (2002). SmartClients: constraint satisfaction as a paradigm for scaleable intelligent information systems. International Journal of Constraints, 7(1), 49–69. CrossRefGoogle Scholar
  38. 38.
    Torrens, M., Weigel, R., & Faltings, B. (1997). Java constraint library: bringing constraints technology on the internet using the Java language. In Workshop notes, constraints and agents, the fourteenth national conference on artificial intelligence (AAAI’97). Merlo Park: AAAI Press. Google Scholar
  39. 39.
    Tversky, A., Sattath, S., & Slovic, P. (1988). Contigent weighting in judgement and choice. Psychology Review, 95, 371–384. CrossRefGoogle Scholar
  40. 40.
    Zhang, J., & Pu, P. (2004). Survey of solving multi-attribute decision problems (Technical Report No. IC/200454). Swiss Federal Institute of Technology in Lausanne (EPFL), Lausanne, Switzerland, June, 2004. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  1. 1.Human Computer Interaction Group, School of Computer and Communication SciencesSwiss Federal Institute of Technology in Lausanne (EPFL)LausanneSwitzerland
  2. 2.Business Administration, Darden Graduate School of BusinessUniversity of VirginiaCharlottesvilleUSA

Personalised recommendations