Advertisement

English Teaching & Learning

, Volume 43, Issue 3, pp 297–315 | Cite as

Validating Research-Abstract Writing Assessment Through Latent Regression Modeling and Rater’s Lenses

  • Ming-Chia LinEmail author
Original Paper

Abstract

This study validates the measure, research-abstract writing assessment (RAWA), with two rating scales of global move of rhetorical purpose and local pattern of language use in applied linguistics (the scale level/score ranging from 0 to 5). The study adopted the embedded design of mixed-methods research that included both the quantitative latent regression model (LRM) for testing how the examinees’ (30 EFL doctoral students, 30 EFL master’s students) RAWA responses can be explained by examinee-group competence, scale-by-level difficulty of two scales, and rater expertise (5 raters); and the qualitative interviews on five raters’ perceptions. The LRM results revealed the scale-level difficulty effect, namely across the scales level 1 and level 5 of the global move being the easiest and the most difficult. The expert raters rated with lower scores. They also adopted the advanced subscales (i.e., content elements, brevity) as criteria and conducted self-monitoring while rating. The findings reveal the sub-competences of research-abstract writing, namely the global move sub-competence of move and content elements and the local pattern sub-competence of language use and brevity. Pedagogically, EFL graduate students should further develop the sub-competences of content element and brevity, once mastering those of move and language use as the basics.

Keywords

Validation of the research-abstract writing assessment (the RAWA) The embedded design of mixed-methods research The latent regression modeling (the LRM) 

英文論文摘要寫作評量效度化之研究

中文摘要

本文驗證一個論文摘要寫作評量, 此評量使用兩個評分表 (整體性修辭文步、局部性語言構式)於應用語言學學門的論文摘要寫作評量之中 。本文使用鑲嵌式設計的混合研究法, 亦即以質性評分者訪談結果(5位評分者)補充量性潛在迴歸模式分析結果 。潛在迴歸模式用於考驗受試者 (碩士生、博士生各30位)的寫作評量作答反映是否可被受試者知能、評分量尺各等級的難度、評分者的專業度三個變項解釋 。模式考驗結果指出評分量尺各等級的難度對受試者反映有解釋效果(如:修辭文步評量表的一分、五分分別為困難度最低、最高的反映); 評分者的專業度有效果, 即專家評分者給分較低 。專家評分者通常採用進階的評分量尺 (內容要件、簡潔性), 作為評分歷程中自我檢視的標準 。這些研究發現揭示論文摘要寫作的次項知能, 即整體性修辭文部知能包含文步使用、內容要件, 局部性語言構式包含語言使用、簡潔性 。教學意涵如下 : 外語研究生應先發展語言和文步使用等基礎知能, 後續再發展內容要件和簡潔性知能 。

關鍵字

論文摘要寫作評量 鑲嵌式設計的混合研究法 潛在迴歸模式 

Notes

Acknowledgments

This research was supported by Ministry of Science and Technology in Taiwan under Grant MOST 103-2410-H-656-003.

Supplementary material

42321_2019_30_MOESM1_ESM.docx (22 kb)
ESM 1 (DOCX 21 kb)

References

  1. 1.
    Adams, R. & Wu, M. (2010). Unidimensional regression model. Retrieved December 1, 2017, from http://www.coe.int/t/dg4/linguistic/Source/CEF-refSupp-SectionH.pdf.
  2. 2.
    AUTHORS (2014).Google Scholar
  3. 3.
    AUTHORS (2015).Google Scholar
  4. 4.
    Cargill, M., Gao, X., Wang, X., & O’Connor, P. (2018). Preparing Chinese graduate students of science facing an international publication requirement for graduation: adapting an intensive workshop approach for early-candidature use. English for Specific Purposes, 52, 13–26.CrossRefGoogle Scholar
  5. 5.
    Chang, P. (2012). Using a stance corpus to learn about effective authorial stance-taking: a textlinguistic approach. ReCALL, 24(2), 209–236.CrossRefGoogle Scholar
  6. 6.
    Cho, D. W. (2009). Science journal paper writing in an EFL context: the case of Korea. English for Specific Purposes, 28, 230–239.CrossRefGoogle Scholar
  7. 7.
    Christensen, K. B. & Kreiner, S. (2004). Testing the fit of latent regression models. Communications in statistics theory and methods, 33(6), 1341–1356.Google Scholar
  8. 8.
    Creswell, J. W. (2014). Research design: qualitative, quantitative, and mixed methods approaches (4th ed.). London: Sage Publications Ltd.Google Scholar
  9. 9.
    Dattalo, P. (2008). Determining sample size: balancing power, precision, and practicality. New York: Oxford University Press.CrossRefGoogle Scholar
  10. 10.
    DeCarlo, L. T. (2002). A latent class extension of signal detection theory, with applications. Multivariate Behavioral Research, 37, 423–451.CrossRefGoogle Scholar
  11. 11.
    Engelhard, G. (2013). Invariant measurement: using Rasch models in the social, behavioral and health sciences. New York: Routledge.CrossRefGoogle Scholar
  12. 12.
    Field, A. (2009). Discovering statistics using SPSS (3rd ed.). London: Sage.Google Scholar
  13. 13.
    Green, A. (2013). Washback in language assessment. International Journal of English Studies, 13(2), 39–51.CrossRefGoogle Scholar
  14. 14.
    Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.CrossRefGoogle Scholar
  15. 15.
    Hamp-Lyons, L. (2011). Writing assessment: shifting issues, new tools, enduring questions. Assessing Writing, 16(1), 3–5.CrossRefGoogle Scholar
  16. 16.
    Hsieh, W. M., & Liou, H. C. (2008). A case study of corpus-informed online academic writing for EFL graduate students. CALICO Journal, 26(1), 28–47.Google Scholar
  17. 17.
    Lallmamode, S. P., Daud, N. M., & Kassim, N. L. A. (2016). Development and initial argument-based validation of a scoring rubric used in the assessment of L2 writing electronic portfolios. Assessing Writing, 30, 44–62.CrossRefGoogle Scholar
  18. 18.
    Leckie, G., & Baird, J. A. (2011). Rater effects on essay scoring: a multilevel analysis of severity drift, central tendency, and rater experience. Journal of Educational Measurement, 48(4), 399–418.CrossRefGoogle Scholar
  19. 19.
    Lee, D., & Swales, J. (2006). A corpus-based EAP course for NNS doctoral students: moving from available specialized corpora to self-compiled corpora. English for Specific Purposes, 25, 56–75.CrossRefGoogle Scholar
  20. 20.
    Linacre, J. M. (2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3, 85–106.Google Scholar
  21. 21.
    Linacre, J. M. (2006). A user’s guide to FACETS: Rasch-model computer program. Retrieved July 30, 2017 from: https://www.winsteps.com/winman/misfitdiagnosis.htm.
  22. 22.
    Liou, H. C., Yang, P. C., & Chang, J. S. (2012). Language supports for journal abstract writing across disciplines. Journal of Computer Assisted Learning, 28(4), 322–335.CrossRefGoogle Scholar
  23. 23.
    Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.CrossRefGoogle Scholar
  24. 24.
    Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: Macmillan.Google Scholar
  25. 25.
    Messick, S. (1995). Validity of psychological assessment: validation of inferences from person’s responses and performance as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.CrossRefGoogle Scholar
  26. 26.
    Sudweeks, R., Reeve, S., & Bradshaw, W. (2005). A comparison of generalizability theory and many-facet Rasch measurement in an analysis of college sophomore writing. Assessing Writing, 9, 239–261.CrossRefGoogle Scholar
  27. 27.
    Swales, J. (1990). Genre analysis: English in academic and research settings. Cambridge, UK: Cambridge University Press.Google Scholar
  28. 28.
    Swales, J., & Feak, C. (2009). Abstracts and the writing of abstracts. Ann Arbor, MI: University of Michigan Press.CrossRefGoogle Scholar
  29. 29.
    Swales, J., & Feak, C. (2012). Academic writing for graduate students: essential tasks and skills (2nd ed.). Ann Arbor, MI: University of Michigan Press.CrossRefGoogle Scholar
  30. 30.
    Wang, J., Engelhard, G., Raczynski, K., Song, T., & Wolf, W. (2017). Evaluating rater accuracy and perception for integrated writing assessments using a mixed-methods approach. Assessing Writing, 33, 36–47.CrossRefGoogle Scholar
  31. 31.
    Wright, B. D., & Masters, G. N. (1982). Rating scale analysis. Chicago: MESA Press.Google Scholar
  32. 32.
    Wright, B. D., & Stone, M. (1979). Best test design. Chicago: MESA Press.Google Scholar
  33. 33.
    Wu, M. L., Adams, R. J., Wilson, M. R., & Haldane, S. A. (2007). ACERConQuest version 2: generalised item response modelling software. Camberwell: Australian Council for Educational Research.Google Scholar
  34. 34.
    Wu, M., Tam, H. P., & Jen, T. H. (2016). Chapter 14 Bayesian IRT models (MML estimation). In M. Wu, H. P. Tam, & T. H. Jen (Eds.), Educational measurement for applied researchers (pp. 261–282). Singapore, SG: Springer.CrossRefGoogle Scholar

Copyright information

© National Taiwan Normal University 2019

Authors and Affiliations

  1. 1.Research Center for Curriculum and InstructionNational Academy for Educational ResearchNew Taipei CityTaiwan, Republic of China

Personalised recommendations