Encyclopedia of Educational Innovation

Living Edition
| Editors: Michael A. Peters, Richard Heraud

Affordances of Digital Simulations to Measure Communicative Success

  • Julie M. SykesEmail author
  • Margaret M. Malone
  • Linda Forrest
  • Ayşenur Sağdıç
Living reference work entry

Latest version View entry history

DOI: https://doi.org/10.1007/978-981-13-2262-4_90-2


Technological innovation and human communication have a complex and entangled past that will, undoubtedly, continue into the future as new tools emerge and people continue to adapt their communication with, around, and through digital tools. Communicative needs influence technological development. Take, for example, the increasing discursive capabilities of text message applications (e.g., added turn-taking features to show real time interaction and create-your-own emojis to express emotion) or the ever-increasing presence of memes, videos, and images in social media applications like Instagram or YouTube. Furthermore, the affordances of digital innovation often provide capabilities that propel theoretical thinking and the ability to work with complex data and delivery systems (Taguchi and Sykes 2013). One such example is the way language is used, analyzed, and taught; using tools to facilitate complex teaching, learning, and assessment experiences that are practical and scalable makes this possible. As a result, transformative models can also be implemented to augment communicative success, especially regarding intercultural communicative competence (ICC) (i.e., the ability to interact with someone of another culture or subculture) and interlanguage pragmatics (ILP) (i.e., the way meaning is communicated and interpreted in language).

Assessing language ability, especially intercultural, pragmatic, and interactional competence (IPIC), is one area in which the concurrent development of transdisciplinary innovation and technological innovation combines. This convergence enables the feasible and meaningful assessment of a comprehensive set of complex language abilities. The IPIC model entails four components – knowledge, analysis, subjectivity, and awareness. The section that follows describes how each of these four components combines to operationalize key dimensions to assess second language interactional, pragmatic, and intercultural competence without minimizing the dynamic nature of interaction. That is, learners can be assessed based on their ability to interact with a number of different interlocutors in numerous contexts using their own preferences while not being required to adhere to a rigid model where there is only one right answer. In addition, this entry describes ways in which digital simulations are ideal for the assessment of complex language abilities by allowing for individual variety and preference while also providing a systematic approach to attaining a learner profile score. The entry concludes with concrete examples practitioners and researchers can use to incorporate such an approach into classroom and large-scale assessment protocols.

Making the Case

Transdisciplinary Innovation in Language Abilities

Communication is most successful when speakers are able to communicate their own meaning and interpret others’ meanings. Messages can be directly encoded via words, or they can be conveyed indirectly through, for example, sequencing, implicature, hints, or gesture. Miscommunication occurs when the intention and interpretation do not match, a notable, common occurrence in multilingual interactions (Taguchi 2012). For example, if a speaker intends to apologize in most varieties of English but does not use an appropriate explanation because explanations are not appropriate in their dominant language, their apology might come across as insincere. Thus, it becomes essential to teach and assess intercultural, pragmatic, and interactional skills as part of language study. Aside from one exception (Roever 2013), the teaching and assessment of intercultural and pragmatic competence are absent from the majority of language classrooms, with no comprehensive assessment measures available to teachers and learners (Roever et al. 2014). A number of factors contribute to this notable absence, including immense language variety across language varieties and across interactional contexts, individual preferences and personalities, and difficulty in assessing a dynamic set of skills that vary based on the interactional contexts (see, e.g., Félix-Brasdefer 2007). Despite these challenges, classroom implementation and valid assessment of ICC and ILP are possible (Taguchi 2015).

To address many of these key challenges, it is fundamental to successfully employ a model which determines success measured as a reflection of learners’ multilingual abilities across a variety of contexts (Sykes 2016). Responsive to both the systemic and dynamic properties of language, the IPIC framework proposed here represents a new direction and presents an innovative method for determining success in multicultural interactions. Drawing on content from 57 theoretical models of intercultural communicative competence, interactional competence, and pragmatic competence, the IPIC model is designed as a synthesis of skills common to a variety of perspectives and deemed most critical for learning interventions and the development of assessment measures – knowledge, analysis, subjectivity, and awareness. The model is unique because it integrates the vital structural component common to many approaches to interlanguage pragmatics while also considering analytical skills, informed learner choice (i.e., subjectivity), and awareness of the perlocutionary force (i.e., consequence) of discourse sequences. The approach provides an innovative means to assess communicative success and the components integral to human interaction, regardless of the context where the interaction is occurring. Furthermore, it moves beyond an approach which privileges the structural elements (i.e., knowledge) of intercultural communicative competence, and it extends that model to the application and interpretation within social contexts.

Knowledge focuses on the ways in which words and structures are employed to engage in interaction. It includes, for example, mastery of semantic formulae (e.g., the greeting sequences needed to say hello), turn-taking mechanisms (e.g., the length of a pause needed to switch speakers), register-appropriate lexicon (e.g., could you as compared to can you), and other fundamental structural components of language (e.g., discourse markers such as well, hmm, let me see). In the communicative scenario of a leave-taking, the knowledge needed would be the expressions used to indicate the need to close a conversation or leave, “good-bye” expressions (tailored for the context), and semantic formulae for the final closing strategies employed.

Analysis refers to the learner’s ability to use the appropriate linguistic strategies, sequencing, and orientation (i.e., directed at the speaker or hearer) based on their intended illocutionary force. Illocutionary force refers to the intended message of the speaker (e.g., a speaker used the phrase It’s raining cats and dogs to indicate that there is a downpour outside or uses profanity with a close friend as a humorous way to build closeness). Using the appropriate utterance to reflect one’s intention requires the ability to analyze potential options and select the appropriate utterances based on the context and interlocutor. In the same leave-taking scenario, analysis includes the ability to properly place a pre-closing (or multiple pre-closings) before the “good-bye” expression, to determine the appropriate time to leave, and the selection of the ultimate leave-taking strategy (e.g., making future plans or expressing gratitude).

Subjectivity, defined by Ishihara and Tarone (2009) as a dynamic approach to learners’ identity and informed decision-making, refers to learners’ ability to articulate why they made the choices they did. Learners might, for example, explain the sequences they use to align with their interlocutors’ expectations (i.e., “I chose a pre-closer first, because I wanted to give some warning I needed to get going”) or might be an explicit and intentional divergence from cultural norms (i.e., “I didn’t want the interaction to drag on, so, even though I knew it was abrupt, I avoided a pre-closer and apologized for needing to leave so abruptly”). The ability to distinguish subjectivity from lack of knowledge is critical for communicative success and the measurement of learners’ abilities.

Finally, awareness refers to the learners’ ability to determine the perlocutionary force (i.e., consequence) of their interaction(s). This might include, for example, the recognition that the interlocutor was becoming tired, so they started the leave-taking sequence or other interaction to respond to the needs of the interlocutor.

The IPIC model offers an organizational and theoretical framework explicitly synthesizing previous models of ICC and ILP to enable learners, even those at the novice level, to engage with the material throughout their learning experience. It also serves as a parallel framework for ability measurement across each of the categories. In order to measure these abilities, learners are asked to engage with a series of simulated scenarios which each entail a number of items related to knowledge, analysis, subjectivity, and awareness. In Part 1, the learner participates in the interaction with a nonplayer character to complete the task. Learners’ responses are scored as related to that context. Then, in Part 2 the learner is asked to reflect on those responses and describe why they said what they said as well as analyze the appropriateness of the interaction. This reflective component is then scored by human raters using a context-specific rubric. Finally, the machine-scored components of the simulation are compiled with the rater scores to produce a learner profile.

Digital Simulations for Innovative Assessment

The measurement of learners’ ICC and ILP abilities requires a system that can deliver and track dynamic human behaviors consistently and responsively. This entails the capability to track abilities in multiple categories simultaneously, adapt to user needs and preferences, integrate retrospective protocols, and be scaled from a small pilot test group to a large number of users. In this light, digital simulations offer an ideal context to facilitate the development and delivery of a large-scale assessment to measure the intercultural, pragmatic, and interactional abilities of language learners. Drawing on the unique affordances of this digital context (see Sykes 2016 for a review), the IPIC measure guides learners through a series of lifelike conversational scenarios with embedded retrospective protocols to measure their language abilities in each of the four areas described above – knowledge, analysis, subjectivity, and awareness. Initially built and piloted in Chinese, English, and Spanish, upon completion, learners receive a report as a profile in which their skills in each of the four areas are scored. This report serves both as an indicator of their performance on the four measures, as well as a report to facilitate the development of additional classroom interventions and suggest additional out-of-class resources for continued development. It is the first of its kind. While still under development, initial pilot data indicate the measure is a valid approach in terms of user perception and learning outcomes, making the measurement of communicative success practical and multidimensional.

The use of digital simulations is ideally suited to the measurement of ICC and ILP abilities for a variety of reasons. First, simulations provide the opportunity for meaningful engagement with the scenarios in which learners find themselves. In doing so, digital simulations accommodate individualized experiences that can also be validated across a large number of learners (see Reinhardt 2019). Secondly, assessment via scenarios is a long-established process, particularly in oral proficiency assessment (Malone and Montee 2010). Such an approach allows the test developer to identify specific domain functions, contexts, and content areas to be targeted (Carroll 2017). Such approaches have been successful in the traditional oral proficiency interview (OPI) and its technological offshoots, such as the computerized oral proficiency interview and OPI-Computer (OPIc) (Malone and Montee 2010). One reason that computer simulations are especially effective is that test takers can be placed in the targeted virtual environment and be situated to interact with different individuals. Apart from a role play scenario, such approaches are inauthentic in face-to-face settings. Thus, computerized simulations allow participants a wide range of choices within the target culture that can allow them to address and show their facility with different contexts and individuals, all of which may require different language and intercultural choices. Furthermore, the simulations can be responsive to learners’ choices, offering an adaptive experience, without limiting the ability to attain a reliable measure. Moreover, utilizing a series of lifelike scenarios, learners can interact with speakers of varying interactional expectations. This facilitates the measurement of abilities across language varieties without privileging one over another, a historical challenge in the measurement of ICC and ILP abilities (Roever et al. 2014). In the digital simulation context, learners’ abilities to interact in culturally appropriate ways can be observed in simulated, but realistic and replicable, scenarios that allow the learners to interact with variety from the target culture based on individual nuances Finally, the scenarios are designed to examine a set of macro-level skills to interface with other linguistic abilities. Integrating retrospective protocols as part of the simulations themselves adds depth to the measurement and allows for the assessment of affective dimensions of human interaction, such as subjectivity and awareness of perlocutionary force (i.e., the resulting consequence of any interaction).

Educational Innovations and Implications

Implications for Classroom Implementation and Large-Scale Assessment

The IPIC project offers notable advancement, both theoretical and technological. The framework can be scaled-down and adapted for practical classroom implementation of explicit teaching of intercultural and pragmatic abilities.
  • Instructors can build lessons based on the four framework dimensions – knowledge, analysis, subjectivity, and awareness. This can start on the first day of class. Take, for example, the case of greetings, often a language function taught in the first week of any beginning sequence. As part of the greetings lesson, an instructor could include the knowledge (e.g., greeting sequences), analysis (e.g., when to use what sequence), subjectivity (e.g., a learner’s choice to greet someone or not), and awareness (e.g., articulating the consequences of the greeting). This adds a dynamic component to the function being studied by asking learners to not only memorize what to say but when and how to say it as related to their own life experience. Then, as learners progress, learning outcomes in each domain can be complexified and made relevant to communicative scenarios with additional complexity and abstraction (i.e., apologizing to a friend when one makes a mistake or does something that bothers the friend).

  • Learners can be given a template with the four areas to investigate a function on their own. For example, if the function is telling stories in the past, learners might be asked to investigate key components such as where to start the sequence of a story, how to show interest in another’s story, the role of overlap (i.e., interruption), and how to determine when others are listening to a story. This allows them to not only learn key linguistic elements of past narration (e.g., temporal markers, past tense verb conjugation), but also the way to use various elements as a competent multilingual speaker. Learners can present this information to the class or build a paper-based simulation task to teach the key features to their peers.

  • Classroom assessment can ask learners to analyze the communicative functions being studied from all four directions. This could include explicit questions in each domain or a small digital or paper-based simulation.

Through the utilization of a research-based framework encompassing a wide variety of theoretical perspectives, teachers and learners can work toward optimal outcomes without digging through the myriad of sometimes conflicting, theoretical models. Utilizing a synthesized heuristic, teachers and students can employ a systematic approach with four key domains – knowledge, analysis, subjectivity, and awareness. Furthermore, in an era of accountability and testing, the IPIC measure enables the consideration of dynamic language skills without reducing the score to the measurement of only lexical recall and production and facility with grammatical structures. Utilizing the affordances offered via innovative digital tools, comprehensive large-scale assessment is feasible with valid and reliable scores that can be utilized in high-stakes contexts, thereby enabling the measurement of communicative success across all language abilities. It can provide legislators, administrators, and teachers the critical scores they need without reducing language abilities to a single score.



  1. Carroll, B. A. (2017). A learning-oriented assessment perspective on scenario-based assessment. Working Papers in Applied Linguistics & TESOL, 17(2), 28–35.Google Scholar
  2. Félix-Brasdefer, C. J. (2007). Pragmatic development in the Spanish as a FL classroom: A cross sectional study of learner requests. Intercultural Pragmatics, 4(2), 253–286.CrossRefGoogle Scholar
  3. Ishihara, N., & Tarone, E. (2009). Emulating and resisting pragmatic norms: Learner subjectivity and pragmatic choice in L2 Japanese. In N. Taguchi (Ed.), Pragmatic competence (pp. 101–128). New York: Mouton de Gruyter.Google Scholar
  4. Malone, M. E., & Montee, M. J. (2010). Oral proficiency assessment: Current approaches and applications for post secondary foreign language programs. Language and Linguistics Compass, 4, 972–986.  https://doi.org/10.1111/j.1749-818X.2010.00246.x.CrossRefGoogle Scholar
  5. Reinhardt, J. (2019). Gameful second and foreign language teaching and learning: Theory, research and practice. New York: Palgrave Macmillan.CrossRefGoogle Scholar
  6. Roever, C. (2013). Assessing pragmatics. The Companion to Language Assessment, 1, 125–139.CrossRefGoogle Scholar
  7. Roever, C., Fraser, C., & Elder, C. (2014). Testing ESL sociopragmatics: Development and validation of a web-based test battery. Frankfurt: Peter Lang.CrossRefGoogle Scholar
  8. Sykes, J. (2016). Technologies for teaching and learning intercultural competence and interlanguage pragmatics. In S. Sauro & C. Chapelle (Eds.), Handbook of technology and second language teaching and learning (pp. 119–133). New York: Wiley.Google Scholar
  9. Taguchi, N. (2012). Context, individual differences, and pragmatic competence. New York/Bristol: Multilingual Matters.CrossRefGoogle Scholar
  10. Taguchi, N. (2015). Instructed pragmatics at a glance: Where instructional studies were, are, and should be going. State-of-the-art article. Language Teaching, 48, 1–50.CrossRefGoogle Scholar
  11. Taguchi, N., & Sykes, J. (2013). Technology in interlanguage pragmatics research and teaching. Amsterdam: John Benjamins Publishing.CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  • Julie M. Sykes
    • 1
    Email author
  • Margaret M. Malone
    • 2
  • Linda Forrest
    • 1
  • Ayşenur Sağdıç
    • 2
  1. 1.University of OregonEugeneUSA
  2. 2.Georgetown UniversityWashingtonUSA

Section editors and affiliations

  • Okim Kang
  • Alyssa Kermad
    • 1
  1. 1.Applied Linguistics, English and Modern Languages DepartmentCalifornia State Polytechnic UniversityPomonaUSA