Multifaceted Rasch Analysis of Paired Oral Tasks for Japanese Learners of English

Abstract

Background and aims: Despite the importance of enhancing and assessing oral interactive ability, few studies have investigated paired oral assessment for Japanese learners of English. This study refines Koizumi et al. (in press), expands the number of paired oral tasks that are calibrated on a logit scale, and examines aspects of validity mainly related to paired oral tasks and raters. Methods: A total of 190 Japanese students from three universities participated in 11 paired oral tasks. Their responses were recorded and evaluated by three raters using a holistic scale. A multifaceted Rasch measurement program, Facets (Linacre 2014), was used. The rating scale model was used to examine the test takers’ abilities, task difficulty, rater severity, and rating scale functions. Structural equation modeling and generalizability theory were also employed. Results and discussions: Results showed a unitary factor structure in the test, with some error correlations between tasks and between raters. We also found that all tasks and raters fit the Rasch model, the rating scale functioned properly, and there was a relatively wide range of tasks in terms of difficulty levels but also a lack of tasks at the higher and lower ends and in between. Results also suggested that large percentages of score variance were explained by persons (test takers), interactions between persons and tasks and between persons and raters, and residuals, and that four tasks with two raters or three tasks with three raters are needed to gain a sufficient reliability of φ = 0.70. As a result, we could provide pieces of validity evidence for the interpretation of the paired oral tasks developed.