Abstract
We evaluated whether a scenario-based assessment structure with a theoretically determined task order has an impact on the writing processes that students execute in responding to an essay task. Students’ writing processes were recorded via keystroke logs. Two testing conditions were compared, one with the original task ordering and the other with reversed task ordering, using more than two dozen variables extracted from keystroke logs. The results showed clear effects of task ordering in the context of scenario-based assessment. The scaffolded form reduced the dependency of performance on general writing fluency. Compared to the non-scaffolded form, students taking the scaffolded form appeared to allocate more cognitive effort to editing and revising, as well as to the higher-level cognitively demanding processes important to the quality of argumentation. Students taking the non-scaffolded form appeared to expend greater effort at the pre-writing stage. Limitations and future directions for research are also discussed.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
The keystrokes are recorded using ETS’ patent-pending keystroke logging system.
- 2.
Researchers had been using these measures like these for many years (e.g., Carlson, 2014; Chase, 1960) but these measures lacked a proper rationale until the axiomatic justification by Pratt. As noted by Pratt’s himself, negative measures can occur. Small negative values could be due to capitalization on chance because both \( {\widehat{\mathrm{r}}}_{\mathrm{p}} \) and \( {\widehat{\beta}}_{\mathrm{p}} \) are sample estimates. Large negative Pratt’s measures can reflect a suppression effect or multicollinearity (Budescu, 1993; Thomas, Hughes, & Zumbo, 1998).
References
Bennett, R. E., Deane, P., & van Rijn, P. W. (2016). From cognitive domain theory to assessment practice. Educational Psychologist, 51, 82–107.
Bruner, J. S. (1978). The role of dialogue in language acquisition. In A. Sinclair, R. J. Jarvelle, & W. J. M. Levelt (Eds.), The child’s concept of language. New York, NY: Springer.
Budescu, D. V. (1993). Dominance analysis: A new approach to the problem of relative importance of predictors in multiple regression. Psychological Bulletin, 114, 542–551.
Carlson, J. E. (2014). A generalization of Pythagoras’s theorem and application to explanations of variance contributions in linear models (Research Report 14–18). Princeton, NJ: Educational Testing Service.
Chase, C. I. (1960). Computation of variance accounted for in multiple correlation. The Journal of Experimental Education, 28, 265–266.
Deane, P. (2014). Using writing process and product features to assess writing quality and explore how those features relate to other literacy tasks (Research Report 14–03). Princeton, NJ: Educational Testing Service.
Deane, P., Sabatini, J. P., Feng, G., Sparks, J., Song, Y., Fowles, M., O’Reilly, T., Jueds, K., Krovetz, R., & Foley, C. (2015). Key practices in the English language arts (ELA): linking learning theory, assessment, and instruction (Research Report 15–17). Princeton, NJ: Educational Testing Service.
Deane, P., & Zhang, M. (2015). Exploring the feasibility of using writing process features to assess text production skills (Research Report 15–26). Princeton, NJ: Educational Testing Service.
Greenfield, P. M., & Smith, J. H. (1976). Structure of communication in early language development. New York, NY: Academic Press.
Guo, H., Deane, P., van Rijn, P. W., Zhang, M., & Bennett, R. E. (2016). Exploring the heavy tailed key-stroke data in writing processes. Manuscript submitted for publication.
Jackson, S. L., Krajcik, J., & Soloway, E. (1998). The design of guided learner-adaptable scaffolding in interactive learning environments. In Proceeding of the SIGCHI conference of human factors in computing systems (pp. 187–194). New York, NY: ACM Press/Addison-Wesley Publishing Co.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York, NY/UK: Macmillan Publishing Co Inc..
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.
Pea, R. D. (2004). The social and technological dimensions of scaffolding and related theoretical concepts for learning, education, and human activity. The Journal of the Learning Sciences, 13, 423–451.
Puntambekar, S., & Hubscher, R. (2005). Tools for scaffolding students in a complex learning environment: What have we gained and what have we missed? Educational Psychologist, 40, 1–12.
Pratt, J. W. (1987). Dividing the indivisible: Using simple symmetry to partition variance explained. In T. Pukilla & S. Duntaneu (Eds.), Proceedings of second tampere conference in statistics (pp. 245–260). Tampere, Finland: University of Tampere.
Sabatini, J. P., O’Reilly, T., Halderman, L., & Bruce, K. (2015). Broadening the scope of reading comprehension using scenario-based assessments: Preliminary findings and challenges. L’ Année Psychologique, 114, 693–723.
Sheehan, K. S., & O’Reilly, T. (2012). The case for scenario-based assessments of reading competency. In J. Sabatini, E. Albro, & T. O’Reilly (Eds.), Reaching an understanding: Innovation in how we view reading assessment. Lanham, MD: R&L Education.
Shepard, L. A. (2005). Linking formative assessment to scaffolding. Educational Leadership, 63(3), 66–71.
Stone, E. (2016). Integrating digital assessment meta-data for psychometric and validity analysis. Manuscript submitted for publication.
Thomas, D. R., Hughes, E., & Zumbo, B. D. (1998). On variable importance in linear regression. Social Indicators Research, 45, 253–275.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. In M. Cole, V. John-Steiner, S. Scribner, & E. Souberman (Eds.). Cambridge, MA: Harvard University Press.
Wiggins, G. (1992). Creating tests worth taking. Educational Leadership, 49(8), 26–33.
Zhang, M. & Deane, P. (2015) Process features in writing: internal structure and incremental value over product features (Research Report 15–27). Princeton, NJ: Educational Testing Service.
Zhang, M., van Rijn, P. W., Deane, P., & Bennett, R. E. (2016). Scenario-based assessments in writing: An experimental study. Manuscript submitted for publication.
Zhang, M., Hao, J., Li, C., & Deane, P. (2016). Classification of writing patterns using keystroke logs. In L. A. van der Ark, D. M. Bolt, W.-C. Wang, J. A. Douglas, & M. Wiberg (Eds.), Quantitative psychology research (pp. 299–314). New York, NY: Springer.
Acknowledgements
We thank James Carlson, Kathy Sheehan, and Jiangang Hao for their technical review of previous versions of this manuscript. We also thank Randy Bennett for his helpful input and Shelby Haberman and Dan McCaffrey for providing advice on the analyses.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
Appendices
Appendix A: CBAL™ Generic Scoring Guide: Discourse-Level Features in a Multi-paragraph Text (Rubric 1)
EXEMPLARY (5)
An EXEMPLARY response meets all of the requirements for a score of 4 but distinguishes itself by skillful use of language, precise expression of ideas, effective sentence structure, and/or effective organization, which work together to control the flow of ideas and enhance the reader’s ease of comprehension.
CLEARLY COMPETENT (4)
A CLEARLY COMPETENT response typically displays the following characteristics:
-
It is adequately structured.
-
Overall, the response is clearly and appropriately organized for the task.
-
Clusters of related ideas are grouped appropriately and divided into sections and paragraphs as needed.
-
Transitions between groups of ideas are signaled appropriately.
-
-
It is coherent.
-
Most new ideas are introduced appropriately.
-
The sequence of sentences leads the reader from one idea to the next with few disorienting gaps or shifts in focus.
-
Connections within and across sentences are made clear where needed by the use of pronouns, conjunctions, subordination, etc.
-
-
It is adequately phrased.
-
Ideas are expressed clearly and concisely.
-
Word choice demonstrates command of an adequate range of vocabulary.
-
Sentences are varied appropriately in length and structure to control focus and emphasis.
-
-
It displays adequate control of Standard Written English
-
Grammar and usage follow SWE conventions, but there may be minor errors.
-
Spelling, punctuation, and capitalization follow SWE conventions, but there may be minor errors.
-
DEVELOPING HIGH (3)
A response in this category displays some competence but differs from Clearly Competent responses in at least one important way, including limited development; inconsistencies in organization; failure to break paragraphs appropriately; occasional tangents; abrupt transitions; wordiness; occasionally unclear phrasing; little sentence variety; frequent and distracting errors in Standard Written English; or relies noticeably on language from the source material.
DEVELOPING LOW (2)
A response in this category differs from Developing High responses because it displays serious problems such as marked underdevelopment; disjointed, list-like organization; paragraphs that proceed in an additive way without a clear overall focus; frequent lapses in cross-sentence coherence; unclear phrasing; excessively simple and repetitive sentence patterns; inaccurate word choices; errors in Standard Written English that often interfere with meaning; or relies substantially on language from the source material.
MINIMAL (1)
A response in this category differs from Developing Low responses because of serious failures such as extreme brevity; a fundamental lack of organization; confusing and often incoherent phrasing; little control of Standard Written English; or can barely develop or express ideas without relying on the source material.
NO CREDIT (0)
Not enough of the student’s own writing for surface-level features to be judged; not written in English; completely off topic; or random keystrokes.
OMIT
Blank.
Copyright by Educational Testing Service, 2013 All rights reserved.
Appendix B: CBAL™ Generic Scoring Guide: Constructing an Argument (Rubric 2)
EXEMPLARY (5)
An EXEMPLARY response meets all of the requirements for a score of 4 and distinguishes itself with such qualities as insightful analysis (recognizing the limits of an argument, identifying possible assumptions and implications of a particular position); intelligent use of claims and evidence to develop a strong argument (including particularly well-chosen examples or a careful rebuttal of opposing points of view); or skillful use of rhetorical devices, phrasing, voice and tone to engage the reader and thus make the argument more persuasive or compelling.
CLEARLY COMPETENT (4)
The response demonstrates a competent grasp of argument construction and the rhetorical demands of the task, by displaying all or most of the following characteristics:
Command of Argument Structure
-
States a clear position on the issue
-
Uses claims and evidence to build a case in support of that position
-
May also consider and address obvious counterarguments
Quality and Development of Argument
-
Makes reasonable claims about the issue
-
Supports claims by citing and explaining relevant reasons and/or examples
-
Is generally accurate in its use of evidence
-
Expresses ideas mainly in the writer’s own words
Awareness of Audience
-
Focuses primarily on content that is appropriate for the target audience
-
Expresses ideas in a tone that is appropriate for the audience and purpose for writing
DEVELOPING HIGH (3)
While a DEVELOPING HIGH response displays some competence, it typically has at least one of the following weaknesses: a vague claim; somewhat unclear, limited, or inaccurate use of evidence; failure to take account of the alternative; noticeable reliance on source language; simplistic reasoning; or occasionally inappropriate content or tone for the audience.
DEVELOPING LOW (2)
A DEVELOPING LOW response displays problems that seriously undermine the writer’s argument, such as a confusing or inconsistent claim; a seriously underdeveloped or unfocused argument; irrelevant confusing, or seriously misused evidence; substantial reliance on source language; an emphasis on opinions or unsupported generalizations rather than reasons and example; or inappropriate content or tone throughout much of the response.
MINIMAL (1)
A MINIMAL response displays little or no ability to construct an argument. For example, there may be no claim, no relevant reasons and examples, no development of an argument, little logical coherence throughout the response, or mainly use of source language.
OFF-TOPIC (0)
Consists entirely of source language, is completely off topic, or consists of random key strokes.
Copyright by Educational Testing Service, 2014 All rights reserved.
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Zhang, M., Zou, D., Wu, A.D., Deane, P., Li, C. (2017). An Investigation of Writing Processes Employed in Scenario-Based Assessment. In: Zumbo, B., Hubley, A. (eds) Understanding and Investigating Response Processes in Validation Research. Social Indicators Research Series, vol 69. Springer, Cham. https://doi.org/10.1007/978-3-319-56129-5_17
Download citation
DOI: https://doi.org/10.1007/978-3-319-56129-5_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-56128-8
Online ISBN: 978-3-319-56129-5
eBook Packages: Social SciencesSocial Sciences (R0)