Investigating the Validity of Using Automated Writing Evaluation in EFL Writing Assessment

Abstract

This study aims to follow an argument-based approach to validation of using automated essay evaluation (AWE) system with the example of Pigai, a Chinese AWE program, in English as a Foreign Language (EFL) writing assessment in China. First, an interpretive argument was developed for its use in the course of College English. Second, three sub-studies were conducted to seek evidence of claims related to score evaluation, score generalization, score explanation, score extrapolation and feedback utilization. Major findings are: (1) Pigai yields scores that are accurate indicators of the quality of a test performance sample; (2) its scores are consistent across tasks in the same form; (3) its scoring features represent the construct of interest to some extent, yet problems of construct under-representation and construct-irrelevant features still exist; (4) its scores are consistent with teachers’ judgments of students’ writing ability; (5) its feedback has a positive impact on students’ development of writing ability, but to some extent. These results reveal that AWE can only be used as a supplement to human evaluation, but can never replace the latter.

This work was supported by the Ministry of Education, P. R. China (Number 17YJC740102) and the Fundamental Research Funds for the Central Universities (Number 2018PY22).

Appendices

Appendix 1: Task 1 (A Technological Invention)

Write an essay of no less than 150 words about a technological invention. Your writing should include four points: 1. An introduction of the invention. 2. Its positive impact on peoples’ life. 3. Its negative impact on people’s life. 4. Your opinion.

Appendix 2: Task 2 (Fame – Good or Evil?)

Write an essay of no less than 150 words on the topic “Fame-Good or Evil?” Your paper should cover the following points: 1. The advantages of being famous. 2. The disadvantages of being famous. 3. Your attitude towards fame.

Appendix 3: The Internet and Our Daily Lives

Write an essay of no less than 150 words on the topic “The Internet and Our Daily Lives”. Your paper should include: 1. Internet is important in our daily lives. 2. Internet has also disadvantages. 3. What shall we do to make better use of Internet?

Appendix 4: A Questionnaire of Teachers’ Attitudes Towards Pigai

• 1. Is Pigai score consistent with your observation of students’ writing ability?

  A. Consistent B. Largely consistent C. Largely inconsistent D. Inconsistent

• 2. Is Pigai’s feedback beneficial to improve students’ writing ability? Why?

• 3. Would you like to let students receive Pigai’s feedback in the future? (Yes/No).