Skip to main content
Log in

Layered evaluation of interactive adaptive systems: framework and formative methods

  • Original Paper
  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

The evaluation of interactive adaptive systems has long been acknowledged to be a complicated and demanding endeavour. Some promising approaches in the recent past have attempted tackling the problem of evaluating adaptivity by “decomposing” and evaluating it in a “piece-wise” manner. Separating the evaluation of different aspects can help to identify problems in the adaptation process. This paper presents a framework that can be used to guide the “layered” evaluation of adaptive systems, and a set of formative methods that have been tailored or specially developed for the evaluation of adaptivity. The proposed framework unifies previous approaches in the literature and has already been used, in various guises, in recent research work. The presented methods are related to the layers in the framework and the stages in the development lifecycle of interactive systems. The paper also discusses practical issues surrounding the employment of the above, and provides a brief overview of complementary and alternative approaches in the literature.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ackerman, M.S., Cranor, L.F., Reagle, J.: Privacy in e-commerce: examining user scenarios and privacy preferences. In: 1st ACM Conference on Electronic Commerce, pp. 1–8. ACM, Denver, CO (1999)

  • Albrecht D., Zuckerman I.: Statistical and probabilistic methods for user modeling. Special issue User Model. User-Adap. Inter. 17(1–2), 1–215 (2007)

    Article  Google Scholar 

  • Anderson, J.R., Boyle, C.F., Yost, G.: The geometry tutor. In: 9th International Joint Conference on Artificial Intelligence, Los Angeles, CA, pp. 1–7. Morgan Kaufmann, San Francisco (1985)

  • Arruabarrena, R., Pérez, T., López-Cu adrado, J., Gutiérrez, J., Vadillo, J.: On evaluating adaptive systems for education. In: 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Malaga, Spain. LNCS, vol. 2347, pp. 363–367. Springer, Berlin (2002)

  • Batliner A., Steidl S., Hacker C., Nöth E.: Private emotions versus social interaction: a data-driven approach towards analysing emotion in speech. User Model. User-Adap. Inter. 18(1-2), 175–206 (2008)

    Article  Google Scholar 

  • Berkovsky S., Kuflik T., Ricci F.: Mediation of user models for enhanced personalization in recommender systems. User Model. User-Adap. Inter. 18(3), 245–286 (2008)

    Article  Google Scholar 

  • Beyer H., Holtzblatt K.: Contextual Design: Defining Customer-Centred Systems. Morgan Kaufmann, San Francisco (1997)

    Google Scholar 

  • Billsus, D., Pazzani, M.: A hybrid user model for news story classification. In: 7th International Conference on User Modeling, Banff, Canada, pp. 98–108. Springer, Vienna (1999)

  • Bontcheva K., Wilks Y.: Tailoring automatically generated hypertext. User Model. User-Adap. Inter. 15(1-2), 135–168 (2005)

    Article  Google Scholar 

  • Boyle C., Encarnacion A.O.: MetaDoc: an adaptive hypertext reading system. User Model. User-Adap. Inter. 4(1), 1–19 (1994)

    Article  Google Scholar 

  • Brown E.J., Brailsford T.J., Fisher T., Moore A.: Evaluating learning style personalization in adaptive systems: quantitative methods and approaches. IEEE Trans. Learn. Technol. 2(1), 10–22 (2009)

    Article  Google Scholar 

  • Browne D., Norman M., Riches D.: Why build adaptive systems. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 15–57. Academic Press, London (1990)

    Google Scholar 

  • Brusilovsky P., Eklund J.: A study of user model based link annotation in educational hypermedia. J. Univ. Comput. Sci. 4(4), 429–448 (1998)

    Google Scholar 

  • Brusilovsky, P., Karagiannidis, C., Sampson, D.: The benefits of layered evaluation of adaptive applications and services. In: 1st Workshop on Empirical Evaluation of Adaptive Systems at UM2001, pp. 1–8. Sonthofen, Germany (2001)

  • Brusilovsky, P., Farzan, R., Ahn, J.: Layered evaluation of adaptive search. In: Workshop on Evaluating Exploratory Search Systems at SIGIR06, pp. 11–13. Seattle, WA (2006)

  • Bull S., Kay J.: Student models that invite the learner in: the SMILI Open learner modelling framework. Int. J. Artif. Intell. Edu. 17(2), 89–120 (2007)

    Google Scholar 

  • Card S.K., Thomas T.P., Newell A.: The Psychology of Human–Computer Interaction. Lawrence Erbaum Associates, London (1983)

    Google Scholar 

  • Carmagnola F., Cena F., Console L., Cortassa O., Gena C., Goy A., Torre I., Toso A., Vernero F.: Tag-based user modeling for social multi-device adaptive guides. User Model. User-Adap. Inter. 18(5), 497–538 (2008)

    Article  Google Scholar 

  • Carmichael D.J., Kay J., Kummerfeld B.: Consistent modelling of users, devices and sensors in a ubiquitous computing environment. User Model. User-Adap. Inter. 15(3-4), 197–234 (2005)

    Article  Google Scholar 

  • Carroll M.J.: Five reasons for scenario-based design. Interact. Comput. 13(1), 43–60 (2000)

    Article  Google Scholar 

  • Cena F., Console L., Gena C., Goy A., Levi G., Modeo S., Torre I.: Integrating heterogeneous adaptation techniques to build a flexible and usable mobile tourist guide. AI Commun. 19(4), 369–384 (2006)

    MATH  MathSciNet  Google Scholar 

  • Chen H.T.: A comprehensive typology for program evaluation. Am. J. Eval. 17(2), 121–130 (1996)

    Article  Google Scholar 

  • Cheverst K., Byun H. E., Fitton D., Sas C., Kray C., Villar N.: Exploring issues of user model transparency and proactive behaviour in an office environment control system. User Model. User-Adap. Inter. 15(3-4), 235–273 (2005)

    Article  Google Scholar 

  • Chickering D.M., Paek T.: Personalizing influence diagrams: applying online learning strategies to dialogue management. User Model. User-Adap. Inter. 17(1-2), 71–91 (2007)

    Article  Google Scholar 

  • Chin D.: Empirical evaluation of user models and user-adapted systems. User Model. User-Adap. Inter. 11(1-2), 181–194 (2001)

    Article  MATH  Google Scholar 

  • Claypool, M., Le, P., Wased, M., Brown, D.: Implicit interest indicators. In: International Conference on Intelligent User Interfaces, pp. 33–40. ACM, Santa Fe, NM (2001)

  • Conati C., MacLaren H.: Empirically building and evaluating a probabilistic model of user affect. User Model. User-Adap. Inter. 19(3), 267–303 (2009)

    Article  Google Scholar 

  • Cooper A.: The Inmates are Running the Asylum. Macmillan, Indianapolis (1999)

    Google Scholar 

  • Cooper, D., Arroyo, I., Woolf, B., Muldner, K., Burleson, W., Christopherson, R.: Sensors model student self concept in the classroom. In: 1st International Conference on User Modeling, Adaptation, and Personalization, Trento, Italy. LNCS, vol. 5535, pp. 30–41. Springer, Berlin (2009)

  • Cramer H., Evers V., Ramlal S., van Someren M., Rutledge L., Stash N., Aroyo L., Wielinga B.: The effects of transparency on trust in and acceptance of a content-based art recommender. User Model. User-Adap. Inter. 18(5), 455–496 (2008)

    Article  Google Scholar 

  • Cronbach L.J., Ambron S.R., Dornbusch S.M., Hess R.D., Hornik R.C., Philips D.C., Walker D.F., Weiner S.S.: Toward Reform of Program Evaluation. Jossey-Bass, San Francisco (1980)

    Google Scholar 

  • Czarkowski, M.: A scrutable adaptive hypertext. PhD Thesis, University of Sydney (2006)

  • Damiano R., Gena C., Lombardo V., Nunnari F., Pizzo A.: A stroll with Carletto: adaptation in drama-based tours with virtual characters. User Model. User-Adap. Inter. 18(5), 417–453 (2008)

    Article  Google Scholar 

  • De Bra, P., Houben, G., Wu, H.: AHAM: a Dexter-based reference model for adaptive hypermedia. In: 10th ACM Conference on Hypertext and Hypermedia, pp. 147–156. ACM, Darmstadt, Germany (1999)

  • de Campos L., Fernández-Luna J., Huete J., Rueda-Morales M.: Managing uncertainty in group recommending processes. User Model. User-Adap. Inter. 19(3), 207–242 (2009)

    Article  Google Scholar 

  • de Rosis, F., Mazzotta, I., Miceli, M., Poggi, I.: Persuasion artifices to promote wellbeing. In: 1st International Conference on Persuasive Technology, Eindhoven, The Netherlands. LNCS, vol. 3962, pp. 84–95. Springer, Berlin (2006)

  • Degemmis M., Lops P., Semeraro G.: A content-collaborative recommender that exploits WordNet-based user profiles for neighborhood formation. User Model. User-Adap. Inter. 17(3), 217–255 (2007)

    Article  Google Scholar 

  • Dey, A.K., Abowd, G.D.: Towards a better understanding of context and context-awareness. In: Workshop on the What, Who, Where, When, and How of Context-Awareness at CHI 2000, pp. 304–307. The Hague, The Netherlands (2000)

  • Dix A., Finlay J., Abowd G., Beale R.: Human Computer Interaction. 2nd edn. Prentice-Hall, Englewood Cliffs (1998)

    Google Scholar 

  • D’Mello S.K., Craig S.D., Sullins J., Graesser A.C.: Predicting affective states through an emote-aloud procedure from AutoTutor’s mixed-initiative dialogue. Int. J. Artif. Intell. Edu. 16(1), 3–28 (2006)

    Google Scholar 

  • D’Mello S.K., Craig S.D., Witherspoon A., McDaniel B., Graesser A.C.: Automatic detection of learner’s affect from conversational cues. User Model. User-Adap. Inter. 18(1-2), 45–80 (2008)

    Article  Google Scholar 

  • Domshlak C., Joachims T.: Efficient and non-parametric reasoning over user preferences. User Model. User-Adap. Inter. 17(1-2), 41–69 (2007)

    Article  Google Scholar 

  • Dumas J.S., Loring B.A.: Moderating Usability Tests: Principles and Practices for Interacting. Morgan Kaufmann, San Francisco (2008)

    Google Scholar 

  • Encarnação, L., Stoev, S.: Application-independent intelligent user support system exploiting action-sequence based user modeling. In: 7th International Conference on User Modeling, pp. 245–254. Banff, Canada (1999)

  • Ericsson, K.A., Simon, H.A.: Protocol analysis: verbal reports as data, revised edition. MIT Press, Cambridge, MA (1993)

  • Forbes-Riley K., Rotaru M., Litman D.J.: The relative impact of student affect on performance models in a spoken dialogue tutoring system. User Model. User-Adap. Inter. 18(1-2), 11–43 (2008)

    Article  Google Scholar 

  • Gena C.: Methods and techniques for the evaluation of user-adaptive systems. Knowl. Eng. Rev. 20(1), 1–37 (2005)

    Article  Google Scholar 

  • Gena, C., Ardissono, L.: Intelligent support to the retrieval of information about hydric resources. In: 3rd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems. LNCS, vol. 3137, pp. 126–135. Springer, Berlin (2004)

  • Gena C., Weibelzahl S.: Usability engineering for the adaptive web. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web: Methods and Strategies of Web Personalization, pp. 720–762. Springer, Berlin (2007)

    Google Scholar 

  • George S., Zukerman I., Niemann M.: Inferences, suppositions and explanatory extensions in argument interpretation. User Model. User-Adap. Inter. 17(5), 439–474 (2007)

    Article  Google Scholar 

  • Glahn, C., Specht, M., Koper, R.: Smart indicators on learning interactions. In: 2nd European Conference on Technology Enhanced Learning, Crete, Greece. LNCS, vol. 4753, pp. 56–70. Springer, Berlin (2007)

  • Goecks, J., Shavlik, J.: Learning users’ interests by unobtrusively observing their normal behavior. In: 5th International Conference on Intelligent User Interfaces, pp. 129–132. ACM, New Orleans, LA (2000)

  • Goren-Bar D., Graziola I., Pianesi F., Zancanaro M.: The influence of personality factors on visitor attitudes towards adaptivity dimensions for mobile museum guides. User Model. User-Adap. Inter. 16(1), 31–62 (2006)

    Article  Google Scholar 

  • Goren-Bar, D., Graziola, I., Rocchi, C., Pianesi, F., Stock, O., Zancanaro, M.: Designing and redesigning an affective interface for an adaptive museum guide. In: 1st International Conference on Affective Computing and Intelligent Interaction, Beijing, China. LNCS, vol. 3784, pp. 939–946. Springer, Berlin (2005)

  • Gould J.D., Lewis C.: Designing for usability: key principles and what designers think. Commun. ACM 28(3), 300–311 (1985)

    Article  Google Scholar 

  • Gould, J., Conti, J., Hovanyecz, T.: Composing letters with a simulated listening typewriter. In: 1st ACM Conference on Human Factors in Computer Systems (CHI), pp. 367–370. ACM, Gaithersburg, MD (1982)

  • Green P.D., Ha E.S., Bullock G.J.: Enough already about “black box” experiments: Studying mediation is more difficult than most scholars suppose. Ann. Am. Acad. Political Soc. Sci. 628(1), 200–208 (2010)

    Google Scholar 

  • Grudin, J., Pruitt, J.: Personas, participatory design, and product development: An infrastructure for engagement. In: Participatory Design Conference, pp. 144–161, ACM, Malmö, Sweden (2002)

  • Guzmán E., Conejo R., Pérez de la Cruz J.-L.: Adaptive testing for hierarchical student models. User Model. User-Adap. Inter. 17(1-2), 119–157 (2007)

    Article  Google Scholar 

  • Herder, E.: Utility-based evaluation of adaptive systems. In: 2nd Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 25–30. Johnstown, PA, USA (2003)

  • Herlocker J.L., Konstan J.A., Terveen L.G., Riedl J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)

    Article  Google Scholar 

  • Hertzum M., Hansen K., Andersen H.H.K.: Scrutinizing usability evaluation: does thinking aloud affect behaviour and mental workload?. Behav. Inf. Technol. 28(2), 165–181 (2009)

    Article  Google Scholar 

  • Hollink V., van Someren M., Wielinga B.: Discovering stages in web navigation for problem-oriented navigation support. User Model. User-Adap. Inter. 17(1-2), 183–214 (2007)

    Article  Google Scholar 

  • Höök K.: Steps to take before intelligent user interfaces become real. Interact. Comput. 12(4), 409–426 (2000)

    Article  Google Scholar 

  • Hoppe, H., Tauber, M., Ziegler, J.: A survey of models and formal description methods in HCI with example applications. In: ESPRIT Project, vol. 385 (1986)

  • Horvitz E., Paek T.: Complementary computing: Policies for transferring callers from dialog systems to human receptionists. User Model. User-Adap. Inter. 17(1-2), 159–182 (2007)

    Article  Google Scholar 

  • Jameson A.: Systems that Adapt to their Users: An Integrative Perspective. Saarland University, Saarbrücken (2001)

    Google Scholar 

  • Jameson A.: Adaptive interfaces and agents. In: Jacko, J.A., Sears, A. (eds) The Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 305–330. L. Erlbaum Associates, Hillsdale (2003)

    Google Scholar 

  • Jameson, A.: User modeling meets usability goals. In: 10th International Conference on User Modeling, Edinburgh, UK. LNAI, vol. 3538, pp. 1–3. Springer, Berlin (2005)

  • Jameson A.: Adaptive user interfaces and agents. 2nd edn. In: Sears, A., Jacko, J. (eds) The Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 433–458. CRC Press, Boca Raton (2008)

    Google Scholar 

  • Jameson A.: Understanding and dealing with usability side effects of intelligent processing. AI Mag. 30(4), 23–40 (2009)

    Google Scholar 

  • Jameson A., Schwarzkopf, E.: Pros and cons of controllability: An empirical study. In: 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Málaga, Spain. LNCS, vol. 2347, pp. 193-202. Springer, Berlin (2002)

  • Kaplan C., Fenwick J., Chen J.: Adaptive hypertext navigation based on user goals and context. User Model. User-Adap. Inter. 3(3), 193–220 (1993)

    Article  Google Scholar 

  • Karagiannidis, C., Sampson, D.: Layered evaluation of adaptive applications and services. In: 1st International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Trento, Italy. LNCS, vol. 1892, pp. 343–346. Springer, Berlin, (2000)

  • Kay, J.: Stereotypes, student models and scrutability. 5th International Conference on Intelligent Tutoring Systems, Montréal, Canada. LNCS, vol. 1839, pp. 19–30. Springer, Berlin, (2000)

  • Kay J.: Learner control. User Model. User-Adap. Inter. 11(1-2), 111–127 (2001)

    Article  MATH  Google Scholar 

  • Knutov E., De Bra P., Pechenizkiy M.: AH—12 years later: a comprehensive survey of adaptive hypermedia methods and techniques. New Rev. Hyperme’d. Multime’d. 15(1), 5–38 (2009)

    Article  Google Scholar 

  • Kobsa A.: Privacy-enhanced web personalization. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web: Methods and Strategies of Web Personalization, pp. 628–670. Springer, Berlin (2007)

    Google Scholar 

  • Koch, N., Wirsing, M.: The Munich reference model for adaptive hypermedia applications. In: 2nd International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Málaga, Spain. LNCS, vol. 2347, pp. 213–222. Springer, Berlin (2002)

  • Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: 14th International Joint Conference on Artificial Intelligence, Montréal, Canada, pp. 1137–1145. Morgan Kaufmann, San Francisco (1995)

  • Kosba E., Dimitrova V., Boyle R.: Adaptive feedback generation to support teachers in web-based distance education. User Model. User-Adap. Inter. 17(4), 379–413 (2007)

    Article  Google Scholar 

  • Krogsæter M., Oppermann R., Thomas G.C.: A user interface integrating adaptability and adaptivity. In: Oppermann, R. (ed.) Adaptive User Support: Ergonomic Design of Manually and Automatically Adaptable Software, pp. 97–125. Lawrence Erlbaum, Hillsdale, NJ (1994)

    Google Scholar 

  • Krueger R., Casey M.: Focus Groups: A Practical Guide for Applied Research. 4th edn. Sage Publications, Los Angeles (2009)

    Google Scholar 

  • Kruppa, M., Aslan, I.: Parallel presentations for heterogeneous user groups—an initial user study. In: 4th International Conference on Intelligent Technologies for Interactive Entertainment, Madonna di Campiglio, Italy. LNAI. vol. 3814, pp. 54–63, Springer, Berlin (2005)

  • Law A., Freer Y., Hunter J., Logie R., McIntosh N., Quinn J.: A comparison of graphical and textual presentations of time series data to support medical decision making in the neonatal intensive care unit. J. Clin. Monitor. Comput. 19(3), 183–194 (2005)

    Article  Google Scholar 

  • Lekakos G., Giaglis G.: A hybrid approach for improving predictive accuracy of collaborative filtering algorithms. User Model. User-Adap. Inter. 17(1-2), 5–40 (2007)

    Article  Google Scholar 

  • Lewis C.: Using the ‘thinking-aloud’ method in cognitive interface design. In: Research Report RC9265. IBM T.J. Watson Research Center, Yorktown Heights, NY (1982)

  • Ley, T., Kump, B., Maas, A., Maiden, N., Albert, D.: Evaluating the adaptation of a learning system before the prototype is ready: A paper-based lab study. In: 1st International Conference on User Modeling, Adaptation, and Personalization, Trento, Italy. LNCS, vol. 5535, pp. 331–336, Springer, Berlin (2009)

  • Limongelli, C., Sciarrone, F., Vaste, G.: LS-Plan: An effective combination of dynamic courseware generation and learning styles in web-based education. In: 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany. LNCS, vol. 5149, pp. 133–142, Springer, Berlin (2008)

  • MacLaren, B., Koedinger, K.: When and why does mastery learning work: instructional experiments with ACT-R “SimStudents”. In: 6th International Conference on Intelligent Tutoring Systems, Biarritz, France, pp. 355–366. Springer, Berlin (2002)

  • Magoulas, G.D., Chen, S.Y., Papanikolaou, K.A.: Integrating layered and heuristic evaluation for adaptive learning environments. In: 2nd Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 5–14, Johnstown, PA (2003)

  • Maguire M.: Methods to support human-centred design. Int. J. Hum. Comput. Stud. 55(4), 587–634 (2001)

    Article  MATH  Google Scholar 

  • Masthoff J.: The evaluation of adaptive systems. In: Patel, N. (ed.) Adaptive Evolutionary Information Systems, pp. 329–347. Idea Group Publishing, London (2002)

    Google Scholar 

  • Masthoff J.: Group modeling: slecting a sequence of television items to suit a group of viewers. User Model. User-Adap. Inter. 14(1), 37–85 (2004)

    Article  Google Scholar 

  • Masthoff, J.: The user as wizard: A method for early involvement in the design and evaluation of adaptive systems. In: 5th Workshop on User-Centred Design and Evaluation of Adaptive Systems at AH06, pp. 460–469. Dublin, Ireland (2006)

  • Masthoff, J.: Automatically constructing good hierarchies: HCI meets AI, unpublished

  • Masthoff J., Gatt A.: In pursuit of satisfaction and the prevention of embarrassment: affective state in group recommender systems. User Model. User-Adap. Inter. 16(3-4), 281–319 (2006)

    Article  Google Scholar 

  • Masthoff, J., Vasconcelos, W.W., Aitken, C., Correa da Silva, F.S.: Agent-based group modelling for ambient intelligence. In: AISB Symposium on Affective Smart Environments, Newcastle, UK (2007)

  • Maulsby, D., Greenberg, S., Mander, R.: Prototyping an intelligent agent through wizard of Oz. In: 10th ACM Conference on Human Factors in Computing Systems, pp. 277–284. ACM, Amsterdam, The Netherlands (1993)

  • McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: How accuracy metrics have hurt recommender systems. In: CHI Work in Progress, Montréal, Canada (2006)

  • Miettinen M., Oulasvirta A.: Predicting time-sharing in mobile interaction. User Model. User-Adap. Inter. 17(5), 475–510 (2007)

    Article  Google Scholar 

  • Millán E., Pérez de la Cruz J.L.: Diagnosis algorithm for student modeling diagnosis and its evaluation. User Model. User-Adap. Inter. 12(2–3), 281–330 (2002)

    Article  MATH  Google Scholar 

  • Mobasher B.: Data mining for web personalization. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The Adaptive Web: Methods and Strategies of Web Personalization, pp. 90–135. Springer, Berlin (2007)

    Google Scholar 

  • : Special issue on data mining for personalization. User Model. User-Adap. Inter. 19(1-2), 1–166 (2009)

    Article  Google Scholar 

  • Moncur, W., Masthoff, J., Reiter, E.: What do you want to know? Investigating the information requirements of patient supporters. In: 21th IEEE International Symposium on Computer-Based Medical Systems, pp. 443–448. IEEE, Jyväskylä, Finland (2008)

  • Murray T.: Formative qualitative evaluation for ‘exploratory’ ITS research. Int. J. Artif. Intell. Educ. 4(2-3), 179–207 (1993)

    Google Scholar 

  • Nguyen, H., Santos, E. Jr.: An evaluation of the accuracy of capturing user intent for information retrieval. In: International Conference on Artificial Intelligence, pp. 341–350. CSREA Press, Las Vegas, NV (2007)

  • Nguyen, H., Masthoff, J., Edwards, P.: Modelling a receiver’s position to persuasive arguments. In: 2nd International Conference on Persuasive Technology, Palo Alto, CA. LNCS, vol. 4744, pp. 271–282. Springer, Berlin (2007)

  • Nielsen J.: Evaluating the thinking-aloud technique for use by computer scientists. In: Hartson H.R., Hix D. (eds.) Advances in Human–Computer interaction, vol.3, pp. 69–82. Ablex, Norwood, NJ (1993)

  • Nielsen J.: Heuristic evaluation. In: Nielsen, J., Mack, R.L. (eds) Usability Inspection Methods, pp. 25–64. John Wiley & Sons, New York (1994a)

    Google Scholar 

  • Nielsen J.: Usability Engineering. 2nd edn. Morgan Kaufmann, San Francisco (1994b)

    Google Scholar 

  • Norman A.D.: How might people interact with agents. Commun. ACM 37(7), 68–71 (1994)

    Article  Google Scholar 

  • Nückles M., Winter A., Wittwer J., Herbert M., Hübner S.: How do experts adapt their explanations to a layperson’s knowledge in asynchronous communication? An experimental study. User Model. User-Adap. Inter. 16(2), 87–127 (2006)

    Article  Google Scholar 

  • Ogata K.: Modern Control Engineering. 5th edn. Prentice Hall, Upper Saddle River, NJ (2009)

    Google Scholar 

  • Ohene-Djan, J.: Ownership transfer via personalisation as a value-adding strategy for web-based education. In: Workshop on Adaptive Systems for Web-Based Education at AH2002, pp. 27–41. Málaga, Spain (2002)

  • Oliver, N., Horvitz, E.: A comparison of HMMs and dynamic bayesian networks for recognizing office activities. In: 10th International Conference on User Modeling, Edinburgh, UK. LNCS, vol. 3538, pp. 199–209. Springer, Berlin (2005)

  • O’Malley, C.E., Draper, S.W., Riley, M.S.: Constructive interaction: A method for studying Human–Computer-human interaction. In: 1st International Conference on Human–Computer Interaction, pp. 269–274. Honolulu, HI (1984)

  • Oppermann R.: Adaptively supported adaptability. Int. J. Hum. Comput. Stud. 40(3), 455–472 (1994)

    Article  Google Scholar 

  • Oppermann R.: Introduction. In: Oppermann, R. (ed.) Adaptive User Support: Ergonomic Design of Manually and Automatically Adaptable Software, pp. 1–13. Lawrence Erlbaum Associates, Hillsdale (1995)

    Google Scholar 

  • Ortigosa, A., Carro, R. M.: The continuous empirical evaluation approach: Evaluating adaptive web-based courses. In: 9th International Conference on User Modeling, Johnstown, PA. LNCS, vol. 2702, pp. 163–167. Springer, Berlin (2003)

  • Paek T., Chickering D.M.: Improving command and control speech recognition on mobile devices: using predictive user models for language modelling. User Model. User-Adap. Inter. 17(1-2), 93–117 (2007)

    Article  Google Scholar 

  • Paramythis, A., Totter, A., Stephanidis, C.: A modular approach to the evaluation of adaptive user interfaces. In: 1st Workshop on Empirical Evaluation of Adaptive Systems at UM2001, pp. 9–24. Sonthofen, Germany (2001)

  • Paramythis, A., Weibelzahl, S.: A decomposition model for the layered evaluation of interactive adaptive systems. In: 10th International Conference on User Modeling, Edinburgh, UK. LNCS, vol. 3538, pp. 438–442 (2005)

  • Person, N.K., Graesser, A.C.: Tutoring research group human or computer? AutoTutor in a bystander Turing test. In: 6th International Conference on Intelligent Tutoring Systems, Biarritz, France. LNCS, vol. 2363, pp. 821–830. Springer, Berlin (2002)

  • Petrelli D., Not E.: User-centred design of flexible hypermedia for a mobile guide: reflections on the HyperAudio experience. User Model. User-Adap. Inter. 15(3-4), 303–338 (2005)

    Article  Google Scholar 

  • Pohl, W.: LaboUr—Machine learning for user modeling. In: 7th International Conference on Human–Computer Interaction, pp. 27–30. Elsevier, Amsterdam (1997)

  • Pohl W.: Logic-based representation and reasoning for user modeling shell systems. User Model. User-Adap. Inter. 9(3), 217–282 (1999)

    Article  MathSciNet  Google Scholar 

  • Popescu, E.: Evaluating the impact of adaptation to learning styles in a web-based educational system. In: 8th International Conference on Web-Based Learning, Aachen, Germany. LNCS, vol. 5686, pp. 343–352. Springer, Berlin (2009)

  • Porayska-Pomsta K., Mavrikis M., Pain H.: Diagnosing and acting on student affect: the tutor’s perspective. User Model. User-Adap. Inter. 18(1-2), 125–173 (2008)

    Article  Google Scholar 

  • Preece J., Rogers Y., Sharp H., Benyon D.: Human–Computer Interaction. Addison-Wesley, Reading (1994)

    Google Scholar 

  • Robson C.: Experiment, Design, and Statistics in Psychology. 3rd edn. Penguin, London (1994)

    Google Scholar 

  • Santos, O., Boticario, J.: Guiding learners in learning management systems through recommendations. In: 4th European Conference on Technology Enhanced Learning, Nice, France. LNCS, vol. 5794, pp. 596–601. Springer, Berlin (2009)

  • Schein, A.I., Popescul, A., Ungar, L.H., Pennock, D.M.: Methods and metrics for cold-start collaborative filtering. In: 25th Annual international ACM SIGIR Conference on Research and Development in Information Retrieval, Tampere, Finland, pp. 253–260. ACM, New York (2002)

  • Schmidt, D., Zukerman, I., Albrecht, D.: Assessing the impact of measurement uncertainty on user models in spatial domains. In: 1st International Conference on User Modeling, Adaptation, and Personalization, Trento, Italy. LNCS, vol. 5535, pp. 210–222. Springer, Berlin (2009)

  • Scriven M.: Produce evaluation. In: Smith, N.L. (ed.) New Techniques for Evaluation, pp. 121–126. Sage, Beverly Hills (1981)

    Google Scholar 

  • Scriven M.: Beyond formative and summative evaluation. In: McLaughlin, G.W., Phillips, D.C. (eds) Evaluation and Education: At Quarter Century, pp. 19–64. University of Chicago Press, Chicago (1991)

    Google Scholar 

  • Scriven M.: Types of evaluation and types of evaluator. Eval. Pract. 17(2), 151–162 (1996)

    Article  Google Scholar 

  • Serna A., Pigot H., Rialle V.: Modeling the progression of Alzheimer’s disease for cognitive assistance in smart homes. User Model. User-Adap. Inter. 17(4), 415–438 (2007)

    Article  Google Scholar 

  • Shneiderman B.: Designing the User Interface: Strategies for Effective Human–Computer Interaction. Addison Wesley, Reading (1998)

    Google Scholar 

  • Sixsmith A.J.: An evaluation of an intelligent home monitoring system. J. Telemed. Telecare 6(2), 63–72 (2000)

    Article  Google Scholar 

  • Spada, D., Sánchez-Montañés, M., Paredes, P., Carro, R.: Towards inferring sequential-global dimension of learning styles from mouse movement patterns. In: 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany. LNCS, vol. 5149, pp. 337–340. Springer, Berlin (2008)

  • Stary, C., Totter, A.: How to integrate concepts of the design and the evaluation of adaptable and adaptive user interfaces. In: 3rd ERCIM Workshop on User Interfaces for All, pp. 68–75. Obernai, France (1997)

  • Stamou S., Ntoulas A.: Search personalization through query and page topical analysis. User Model. User-Adap. Inter. 19(1), 5–33 (2009)

    Article  Google Scholar 

  • Stock O., Zancanaro M.: PEACH—Intelligent Interfaces for Museum Visits. Springer, Berlin (2007)

    Book  Google Scholar 

  • Stock O., Zancanaro M., Busetta P., Callaway C., Krüger A., Kruppa M., Kuflik T. et al.: Adaptive, intelligent presentation of information for the museum visitor in PEACH. User Model. User-Adap. Inter. 17(3), 257–304 (2007)

    Article  Google Scholar 

  • Suebnukarn S., Haddawy P.: Modeling individual and collaborative problem-solving in medical problem-based learning. User Model. User-Adap. Inter. 16(3-4), 211–248 (2006)

    Article  Google Scholar 

  • Tarpin-Bernard, F., Marfisi-Schottman, I., Habieb-Mammar, H.: AnAmeter: The first steps to evaluating adaptation. In: 6th Workshop on User-Centred Design and Evaluation of Adaptive Systems at UMAP2009, pp. 11–20. CEUR, Trento, Italy (2009)

  • Tintarev, N., Masthoff, J.: Effective explanations of recommendations: User-centered design. In: ACM conference on Recommender systems, pp. 153–156. ACM, Minneapolis, MN (2007)

  • Tintarev, N., Masthoff, J.: The effectiveness of personalized movie explanations: An experiment using commercial meta-data. In: 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany. LNCS, vol. 5149, pp. 204–213. Springer, Berlin (2008)

  • Tintarev, N., Masthoff, J.: Evaluating recommender explanations: problems experienced and lessons learned for the evaluation of adaptive systems. In: 6th Workshop on User Centered Design and Evaluation at UMAP09, pp. 54–63. CEUR, Trento, Italy (2009)

  • Tobar, C.M.: Yet another evaluation framework. In: Workshop on Empirical Evaluation of Adaptive Systems at UM2003, pp. 15–24. Johnstown, PA (2003)

  • Totterdell P., Boyle E.: The evaluation of adaptive systems. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 161–194. Academic Press, London (1990)

    Google Scholar 

  • Totterdell P., Rautenbach P.: Adaptation as a problem of design. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 61–84. Academic Press, London (1990)

    Google Scholar 

  • Totterdell P., Rautenbach P., Wilkinson A., Anderson S.: Adaptive interface techniques. In: Browne, D., Totterdell, P., Norman, M. (eds) Adaptive User Interfaces, pp. 131–160. Academic Press, London (1990)

    Google Scholar 

  • Trewin, S.: Configuration agents, control and privacy. In: ACM Conference on Universal Usability, pp. 9–16. ACM, Arlington, VA (2000)

  • Turing A.: Computing machinery and intelligence. Mind 59, 433–460 (1950)

    Article  MathSciNet  Google Scholar 

  • van Barneveld, J., van Setten, M.: Involving users in the design of user interfaces for TV recommender systems. In: 3rd Workshop on Personalization in Future TV at UM03, Johnstown, PA (2003)

  • van den Haak M.J., de Jong M.D.T., Schellens P.J.: Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behav. Inf. Technol. 22(5), 339–351 (2003)

    Article  Google Scholar 

  • van den Haak M.J., de Jong M.D.T., Schellens P.J.: Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison. Interact. Comput. 16(6), 1153–1170 (2004)

    Article  Google Scholar 

  • van Velsen L., van der Geest T., Klaassen R., Steehouder M.: User-centered evaluation of adaptive and adaptable systems: a literature review. Knowl. Eng. Rev. 23(3), 261–281 (2008)

    Google Scholar 

  • VanLehn, K., Niu, Z., Siler, S., Gertner, A.S.: Student modeling from conventional test data: a Bayesian approach without priors. In: 5th International Conference on Intelligent Tutoring Systems, Montréal, Canada. LNCS, vol. 1452, pp. 434–443. Springer, Berlin (1998)

  • Walker E., Rummel N., Koedinger K.R.: CTRL: A research framework for providing adaptive collaborative learning support. User Model. User-Adap. Inter. 19(5), 387–431 (2009)

    Article  Google Scholar 

  • Wang, Y., Chen, Z., Kobsa, A.: A collection and systematization of international privacy laws, with special consideration of internationally operating personalized websites. http://www.ics.uci.edu/~kobsa/privacy (2006)

  • Weber, G., Specht, M.: User modeling and adaptive navigation support in WWW-based tutoring systems. In: 6th International Conference on User Modeling, Chia Laguna, Italy, pp. 289–300. Springer, Vienna (1997)

  • Weibelzahl, S.: Evaluation of adaptive systems. In: 8th International Conference on User Modeling. LNCS, vol. 2109, pp. 292–294. Springer, Berlin (2001)

  • Weibelzahl, S.: Evaluation of adaptive systems. PhD Thesis, University of Trier, Germany (2003)

  • Weibelzahl, S.: Problems and pitfalls in evaluating adaptive systems. In: 4th Workshop on the Evaluation of Adaptive Systems at UM’05, pp. 57–66. Edinburgh, UK (2005)

  • Weibelzahl, S., Weber, G.: Evaluating the inference mechanism of adaptive learning systems. In: 9th International Conference of User Modeling, Johnstown, PA. LNCS, vol. 2702, pp. 154–168. Springer, Berlin (2003)

  • Wharton C., Rieman J., Lewis C., Polson P.: The cognitive walkthrough method: A practitioner’s guide. In: Nielsen, J., Mack, R.L. (eds) Usability Inspection Methods, pp. 105–141. John Wiley & Sons, New York (1994)

    Google Scholar 

  • Wilson J., Rosenberg D.: Rapid prototyping for user interface design. In: Helander, M. (ed.) Handbook of Human–Computer Interaction, pp. 859–875. Elsevier, Amsterdam (1988)

    Google Scholar 

  • Winter, S., Wagner, S., Deissenboeck, F.: A comprehensive model of usability. In: Engineering Interactive Systems Conference. LNCS, vol. 4940, pp. 106–122. Springer, Berlin (2008)

  • Witten I.A., Frank E.: Data Mining: Practical Machine Learning Tools and Techniques. 2nd edn. Morgan Kaufmann, Amsterdam (2005)

    MATH  Google Scholar 

  • Yang, D., Huo, H.: Assessment on the adaptivity of adaptive systems. In: International Conference on Management of e-Commerce and e-Government, pp. 437-440. IEEE, Nanchang, China (2008)

  • Yudelson M., Medvedeva O., Crowley R.: A multifactor approach to student model evaluation. User Model. User-Adap. Inter. 18(4), 349–382 (2008)

    Article  Google Scholar 

  • Zancanaro, M., Kuflik, T., Boger, Z., Goren-Bar, D., Goldwasser, D.: Analyzing museum visitors’ behavior patterns. In: 11th International Conference on User Modeling, Corfu, Greece. LNCS, vol. 4511, pp. 238–246. Springer, Berlin (2007)

  • Zaslow, J.: If TiVo thinks you are gay, here’s how to set it straight. Wall Street J. A 1 (Nov 26, 2002)

  • Zhang, T., Rau, P., Salvendy, G.: Developing instrument for handset usability evaluation: A survey study. In: 12th International Conference on Human–Computer Interaction, Beijing, China. LNCS, vol. 4550, pp. 662–671. Springer, Berlin (2007)

  • Ziegler J., Bullinger J.H.: Formal models and techniques in Human–Computer interaction. In: Shackel, B., Richardson, S.J. (eds) Human Factors for Informatics Usability, pp. 183–206. Cambridge University Press, Cambridge (1991)

    Google Scholar 

  • Ziegler, C.-N., McNee, S.M., Konstan, J.A., Lausen, G.: Improving recommendation lists through topic diversification. In: 14 th International World Wide Web Conference, pp. 22–32. ACM, Chiba, Japan (2005)

  • Zimmermann A., Lorenz A.: LISTEN: A user-adaptive audio-augmented museum guide. User Model. User-Adap. Inter. 18(5), 389–416 (2008)

    Article  Google Scholar 

  • Zimmermann A., Specht M., Lorenz A.: Personalization and context management. User Model. User-Adap. Inter. 15(3-4), 275–302 (2005)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Alexandros Paramythis.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Paramythis, A., Weibelzahl, S. & Masthoff, J. Layered evaluation of interactive adaptive systems: framework and formative methods. User Model User-Adap Inter 20, 383–453 (2010). https://doi.org/10.1007/s11257-010-9082-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-010-9082-4

Keywords

Navigation