The evaluation of a social adaptive website for cultural events

  • Cristina Gena
  • Federica Cena
  • Fabiana Vernero
  • Pierluigi Grillo
Original Paper


In this paper, we present an evaluation of a social adaptive website in the domain of cultural events, iCITY DSA, which provides information about cultural resources and events that promote the cultural heritage in the city of Turin. Using this evaluation, our objective was to investigate the actual usage of a social adaptive website, in an effort to discover the real behavior of users, the unforeseen correlations among user actions and the consequent interactive behavior, the accuracy of both system and social recommendations and their impact on the users themselves, and the role of tagging in the user modeling process. The major contributions of the paper are manifold: insights into user interactions with social adaptive systems; guidelines for future designs; evaluation of the tagging activity and tag meanings in relation to the application domain and thus their impact on the representation of the user model; and a demonstration of how a combination and interplay of evaluation methodologies (e.g., quantitative and qualitative) can enhance our comprehension of evaluation data.


Evaluation Social adaptive systems Tag-based user model Cultural events Social recommenders 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Abel, F., Herder, E., Houben, G.-J., Henze, N., Krause, D.: Cross-system user modeling and personalization on the social web. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)Google Scholar
  2. Bakeman R., Gottman J.M.: Observing Interaction: An Introduction to Sequential Analysis. Cambridge University Press, Cambridge (1997)CrossRefGoogle Scholar
  3. Barker T., Jones S., Britton C., Messer D.: The use of a co-operative student model of learner characteristics to configure a multimedia application. User Model. User-Adapt. Interact. 12(2–3), 207–241 (2002)zbMATHCrossRefGoogle Scholar
  4. Brusilovsky P.: Methods and techniques of adaptive hypermedia. User Model. User-Adapt. Interact. 6(2–3), 87–129 (1996)CrossRefGoogle Scholar
  5. Brusilovsky, P., Karagiannidis, P., Sampson, D.: The benefits of layered evaluation of adaptive applications and services. In: Weibelzahl, D.N.C.S. (ed.) Empirical Evaluation of Adaptive Systems, Proceedings of Workshop At the Eighth International Conference on User Modeling, UM2001, pp. 1–8 (2001)Google Scholar
  6. Brusilovsky, P., Chavan, G., Farzan, R.: Social adaptive navigation support for open corpus electronic textbooks. In: DeBra, P., Nejdl, W. (eds.) Proceedings of Adaptive Hypermedia and Adaptive Web-Based Systems, AH04, vol. 3137 of Lecture Notes in Computer Science, pp. 176–189. Springer, Berlin (2004)Google Scholar
  7. Carmagnola F., Cena F., Console L., Cortassa O., Gena C., Goy A., Torre I., Toso A., Vernero F.: Tag-based user modeling for social multi-device adaptive guides. User Model. User-Adapt. Interact. 18(5), 497–538 (2008)CrossRefGoogle Scholar
  8. Carmagnola, F., Cena, F., Console, L., Grillo, P., Perrero, M., Simeoni, R., Vernero, F.: Supporting content discovery and organization in networks of contents and users. Multimedia Syst. 17, 199–218 (2011a). ISSN 0942-4962, http://10.1007/s00530-010-0219-4
  9. Carmagnola, F., Cena, F., Gena, C.: User model interoperability: a survey. User Model. User-Adapt. Interact. 21, 285–331 (2011b). ISSN 0924-1868. http://10.1007/s11257-011-9097-5
  10. Cena, F., Gena, C., Modeo, S.: How to communicate recommendations? Evaluation of an adaptive annotation technique. In: Costabile, M.F., Paternò, F. (eds.) INTERACT, vol. 3585 of Lecture Notes in Computer Science, pp. 1030–1033. Springer, Berlin (2005). ISBN 3-540-28943-7Google Scholar
  11. Cena, F., Carmagnola, F., Cortassa, O., Gena, C., Wang, Y., Stash, N., Aroyo, L.: Tag interoperability in cultural web-based applications. In: Brusilovsky, P., Davis, H.C. (eds.) Hypertext, pp. 221–222. ACM, Pittsburgh (2008). ISBN 978-1-59593-985-2Google Scholar
  12. Chin, D.N.: Empirical evaluation of user models and user-adapted systems. User Model. User-Adapt. Interact. 11(1–2), 181–194 (2001). ISSN 0924-1868Google Scholar
  13. Damiano R., Gena C., Lombardo V., Nunnari F., Pizzo A.: A stroll with carletto: adaptation in drama-based tours with virtual characters. User Model. User-Adapt. Interact. 18(5), 417–453 (2008)CrossRefGoogle Scholar
  14. deGemmis, M., Lops, P., Semeraro, G., Basile, P.: Integrating tags in a semantic content-based recommender. In: Proceedings of the 2008 ACM conference on Recommender systems, RecSys ’08, pp. 163–170. ACM, New York (2008). ISBN 978-1-60558-093-7.
  15. Diederich, J., Iofciu, T.: Finding communities of practice from user profiles based on folksonomies. In: Proceedings of the 1st International Workshop on Building Technology Enhanced Learning solutions for Communities of Practice (TEL-CoPs’06), October 2006Google Scholar
  16. Fidel, R., Crandall, M.: Users’ perception of the performance of a filtering system. In: Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’97, pp. 198–205. ACM, New York (1997). ISBN 0-89791-836-3.
  17. Freyne, J., Berkovsky, S., Daly, E.M., Geyer, W.: Social networking feeds: recommending items of interest. In: Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys’10, pp. 277–280. ACM, New York (2010). ISBN 978-1-60558-906-0.
  18. Gena C.: Methods and techniques for the evaluation of user-adaptive systems. Knowl. Eng. Rev. 20(1), 1–37 (2005)CrossRefGoogle Scholar
  19. Gena, C., Weibelzahl, S.: Usability engineering for the adaptive web. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web, vol. 4321 of Lecture Notes in Computer Science, pp. 720–762. Springer, Berlin (2007). ISBN 978-3-540-72078-2Google Scholar
  20. Good, N., Schafer, J.B., Konstan, J.A., Borchers, A., Sarwar, B.M., Herlocker, J.L., Riedl, J.: Combining collaborative filtering with personal agents for better recommendations. In: Proceedings of the Sixteenth National Conference on Artificial Intelligence and Eleventh Conference on Innovative Applications of Artificial Intelligence, AAAI/IAAI 1999, pp. 439–446, 18–22 July 1999, Orlando, Florida. AAAI Press/The MIT Press, Menlo Park/Cambridge (1999)Google Scholar
  21. Guy, I., Zwerdling, N., Carmel, D., Ronen, I., Uziel, E., Yogev, S., Ofek-Koifman, S.: Personalized recommendation of social software items based on social relations. In: RecSys ’09: Proceedings of the third ACM Conference on Recommender Systems, pp. 53–60. ACM, New York (2009). ISBN 978-1-60558-435-5. http://10.1145/1639714.1639725.
  22. Herlocker, J.L., Konstan, J.A., Terveen, L., Riedl, J.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004). ISSN 1046-8188. http://10.1145/963770.963772.Google Scholar
  23. Höök, K.:Evaluating the utility and usability of an adaptive hypermedia system. In: Proceedings of International Conference on Intelligent User Interfaces, IUI97, pp. 179–186. ACM Press, New York (1997). http://10.1145/238218.238320.
  24. Jameson, A.: Adaptive interfaces and agents. In: The human-computer interaction handbook: fundamentals, Evolving Technologies and Emerging Applications, pp. 305–330. Lawrence Erlbaum Associates Inc., Hillsdale (2003). ISBN 0-8058-3838-4.Google Scholar
  25. Karagiannidis, C., Sampson, D.G. : Layered evaluation of adaptive applications and services. In: Proceedings of the International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, AH00, pp. 343–346. Springer, London (2000). ISBN 3-540-67910-3Google Scholar
  26. Kay, J.: Scrutable adaptation: Because we can and must. In: Wade, V.P., Ashman, H., Smyth, B. (eds.) AH, vol. 4018 of Lecture Notes in Computer Science, pp. 11–19. Springer, Berlin (2006). ISBN 3-540-34696-1Google Scholar
  27. Keppel, G., Saufley, W.H., Tokunaga, H.: Introduction to Design and Analysis: A Student’s Handbook. W.H. Freeman and Co., New York (1998)Google Scholar
  28. Kim, H.-N., ElSaddik, A.: Exploring social tagging for personalized community recommendations. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)Google Scholar
  29. Kobsa A., Koenemann J., Pohl W.: Personalized hypermedia presentation techniques for improving online customer relationship. Knowl. Eng. Rev. 16(2), 111–115 (2001)zbMATHCrossRefGoogle Scholar
  30. Kyriacou, D., Davis, H.C., Tiropanis, T.: A (multi‘domain’sional) scrutable user modelling infrastructure for enriching lifelong user modelling. In: Proceedings of Lifelong User Modelling Workshop AT Conference UMAP 2009, Trento, pp. 46–54, 2009Google Scholar
  31. Loizou, S.K., Dimitrova, V.: Adaptive notifications to support knowledge sharing in close-knit virtual communities. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)Google Scholar
  32. Marlow, C., Naaman, M., Boyd, D., Davis, M.: Ht06, tagging paper, taxonomy, flickr, academic article, to read. In: 17th Conference on Hypertext and Hypermedia HYPERTEXT ’06, pp. 31–40. ACM, New York (2006)Google Scholar
  33. McLaughlin, M.R., Herlocker, J.L.: A collaborative filtering algorithm and evaluation metric that accurately model the user experience. In: Sanderson, M., Jrvelin, K., Allan, J., Bruza, P. (eds.) SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, 25–29 July 2004, pp. 329–336. ACM, New York (2004)Google Scholar
  34. Michlmayr, E., Cayzer, S.: Learning user profiles from tagging data and leveraging them for personal(ized) information access. In: Proceedings of the Workshop on Tagging and Metadata for Social Information Organization, 16th International World Wide Web Conference, 2007Google Scholar
  35. Nakamoto, R.Y., Nakajima, S., Miyazaki, J., Uemura, S., Kato, H., Inagaki, Y.: Reasonable tag-based collaborative filtering for social tagging systems. In: Proceedings of the 2nd ACM Workshop on Information Credibility on the Web, WICOW ’08, pp. 11–18. ACM, New York (2008). ISBN 978-1-60558-259-7. doi: 10.1145/1458527.1458533
  36. Nauerz, A., Bakalov, F., Welsch, M., Konig-Ries, B.: New tagging paradigms for content recommendation in web 2.0 portals. In: Proceedings of International Workshop on Adaptation and Personalization for Web 2.0 (AP-WEB 2.0 2009), pp. 143–147, 2009Google Scholar
  37. Nielsen, J.: Risks of quantitative studies. Accessed Jan 2009, March 2004
  38. Niu, W.T., Kay, J.: Pervasive personalisation of location information: Personalised context ontology. In: Proceedings of the 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, AH ’08, pp. 143–152. Springer, Berlin (2008). ISBN 978-3-540-70984-8Google Scholar
  39. Nonnecke, B., Preece, J., Andrews, D.: What lurkers and posters think of each other. In: Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS’04)—Track 7—vol. 7, HICSS ’04, p. 70195.1. IEEE Computer Society, Washington (2004). ISBN 0-7695-2056-1.
  40. Paramythis, A., Weibelzahl, S.: A decomposition model for the layered evaluation of interactive adaptive systems. In: User Modeling, vol. 3538 of Lecture Notes in Computer Science, pp. 438–442. Springer, Berlin (2005)Google Scholar
  41. Paramythis, A., Totter, P., Stephanidis, P.: A modular approach to the evaluation of adaptive user interface. In: Weibelzahl, S., Chin, D., Weber, G. (eds.) Proceedings of the First Workshop on Empirical Evaluation of Adaptive Systems, pp. 9–24, Sonthofen, Germany (2001)Google Scholar
  42. Paramythis, A., Weibelzahl, S., Masthoff, J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Model. User-Adapt. Interact. 20, 383–453 (2010). ISSN 0924-1868. http://10.1007/s11257-010-9082-4
  43. Pirolli, P., Kairam, S.: A knowledge-tracing model of learning from a social tagging system. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)Google Scholar
  44. Preece J.J., Nonnecke B., Andrews D.: The top five reasons for lurking: improving community experiences for everyone. Comput. Hum. Behav. 20(2), 201–223 (2004)CrossRefGoogle Scholar
  45. Prieur, C., Cardon, D., Beuscart, J.-S., Pissard, N., Pons, P.: The strength of weak cooperation: a case study on flickr. CoRR. abs/0802.2317 (2008)Google Scholar
  46. Rizzo, P., Lee, H., Shaw, E., Johnson, W.L., Wang, N., Mayer, R.E.: A semi-automated wizard of oz interface for modeling tutorial strategies. In: Proceedings of International Conference on User Modeling, UM 2005, pp. 174–178. doi: 10.1007/11527886
  47. Sarwar, B., Karypis, G., Konstan, J., Reidl, J.: Item-based collaborative filtering recommendation algorithms. In: Proceedings of the 10th International Conference on World Wide Web, pp. 285–295. ACM Press, New York (2001)Google Scholar
  48. Shapira, B., Rokach, L., Freilichman, S.: Utilizing facebook single and cross domain data for recommendation systems. Special issue on Personalization in SocialWeb-Based Systems, User Modeling and User-Adapted Interaction, 23(2–3) (2013)Google Scholar
  49. Shepitsen, A., Gemmell, J., Mobasher, B., Burke, R.: Personalized recommendation in social tagging systems using hierarchical clustering. In: Proceedings of the 2008 ACM conference on Recommender Systems, RecSys ’08, pp. 259–266. ACM, New York (2008). ISBN 978-1-60558-093-7.
  50. Sinha, R.R., Swearingen, K.: Comparing recommendations made by online systems and friends. In: DELOS Workshop: Personalisation and Recommender Systems in Digital Libraries, 2001Google Scholar
  51. Strauss, A., Corbin, J.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. SAGE Publications, Thousand Oaks (1998). ISBN 0803959400Google Scholar
  52. Szomszor, M., Alani, H., Cantador, I., O’Hara, K., Shadbolt, N.: Semantic modelling of user interests based on cross-folksonomy analysis. In: Sheth, A.P., Staab, S., Dean, M., Paolucci, M., Maynard, D., Finin, T.W., Thirunarayan, K. (eds.) International Semantic Web Conference, vol. 5318 of Lecture Notes in Computer Science, pp. 632–648. Springer, Berlin (2008). ISBN 978-3-540-88563-4Google Scholar
  53. Totterdell, P., Boyle, E.: Adaptive user interfaces. In: Browne, D., Totterdell, A., Norman, D. (eds.) The Evaluation of Adaptive Systems, pp. 161–194. Academic Press, London (1990)Google Scholar
  54. van Setten, M., Brussee, R., van Vliet, H., Gazendam, L., van Houten, Y., Veenstra, M.: On the importance of “Who Tagged What”. In: Workshop on the Social Navigation and Community Based Adaptation Technologies, pp. 552–561, Dublin, Ireland, 2006.
  55. Vig, J., Sen, S., Riedl, J.: Navigating the tag genome. In: Pu, P., Pazzani, M.J., André, E., Riecken, D. (eds.) IUI, pp. 93–102. ACM, New York (2011). ISBN 978-1-4503-0419-1Google Scholar
  56. Wang Y., Stash N., Aroyo L., Gorgels P., Rutledge L., Schreiber G.: Recommendations based on semantically enriched museum collections. J. Web Semant. 6(4), 283–290 (2008)CrossRefGoogle Scholar
  57. Wang, Y., Zhang, J., Vassileva, J.:Personalized recommendation of integrated social data across social networking sites. In: Proceedings of SASweb Workshop, at UMAP 2010, pp. 19–30, 2010Google Scholar
  58. Weibelzahl, S.: Evaluation of adaptive systems. In: User Modeling, vol. 2109 of Lecture Notes in Computer Science, pp. 292–294. Springer, Berlin (2001)Google Scholar
  59. Weibelzahl, S.: Evaluation of adaptive systems. Dissertation. University of Trier, Trier (2003)Google Scholar
  60. Weibelzahl, S., Lauer, C.:Framework for the evaluation of adaptive CBR-systems. In: Reimer, U., Schmitt, S., Vollrath, I. (eds.) Proceedings of the 9th German Workshop on Case-Based Reasoning (GWCBR01), Aachen: Shakerm, pp. 254–263 (2001)Google Scholar
  61. Weibelzahl, S., Weber, G.: A database of empirical evaluations of adaptive systems. In: Proceedings of Workshop Lernen–Lehren–Wissen–Adaptivity (LLWA 01), Research Report in Computer Science Nr. 763, University of Dortmund, pp. 302–306, 2001Google Scholar
  62. Zanardi, V., Capra, L.: Social ranking: uncovering relevant content using tag-based recommender systems. In: Pu, P., Bridge, D.G., Mobasher, B., Ricci, F. (eds.) Proceedings of International Conference on Recommender Systems, RecSys08, pp. 51–58. ACM, New York (2008). ISBN 978-1-60558-093-7Google Scholar
  63. Zhen, Y., Li, W.-J., Yeung, D.-Y.: Tagicofi: tag informed collaborative filtering. In: Bergman, L.D., Tuzhilin, A., Burke, R.D., Felfernig, A., Schmidt-Thieme, L. (eds.) RecSys, pp. 69–76. ACM, New York (2009). ISBN 978-1-60558-435-5Google Scholar
  64. Zimmermann, A., Lorenz, A.: Listen: a user-adaptive audio-augmented museum guide. User Model. User-Adapt. Interact. 18, 389–416 (2008). ISSN 0924-1868. doi: 10.1007/s11257-008-9049-x.

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  • Cristina Gena
    • 1
  • Federica Cena
    • 1
  • Fabiana Vernero
    • 1
  • Pierluigi Grillo
    • 1
  1. 1.Dipartimento di InformaticaUniversità di TorinoTorinoItaly

Personalised recommendations