Evaluating the Sentence Form Test as a Test of English Writing for Students in Three Japanese Universities

Abstract

The current project was a follow-up to PROMS presentations in 2014 and 2015, both of which pointed to the potential value of “indirect,” objective writing tests over and above “direct” ratings of writing, for English entrance examinations, as well as lower stakes placement tests for writing classes in Japanese universities. The 2014 results showed that the entrance examination essay rating process of one EAP department in a Japanese university, despite adherence to standard rating procedures, was fraught with difficulties, including unpredictable as well as too limited use of the rubric by raters, and rubric flaws, including unclear definition of score categories. The 2015 results focused on an “indirect” writing test, the Sentence Form Test (SFT), a multiple-choice test of writing knowledge, in particular, correct sentence form. Results showed that the fit, difficulty, and reliability of the SFT were acceptable, and SFT scores also demonstrated criterion-related validity, as shown by their correlations with both analytic and holistic essay ratings. For the current follow-up study, a larger number of students from three Japanese universities were asked to complete the SFT, in order to examine whether it would work equally well as a test for female and male students from different universities, in a variety of programs of study, and of a wider range of English proficiency. Results showed that the SFT performed well overall in assessing students, though more test items were needed for students at the lower proficiency levels. In order to examine the character of the test in more detail, a sample of students with higher levels of English proficiency was drawn from the larger sample, in order to enhance test targeting. Results showed that the SFT demonstrated both unidimensionality and person-invariant item calibration.