Skip to main content
SpringerLink
Log in

The Sixth Validation Study: Assessing the Ease of Use in the Environment and Markers’ Acceptance of Onscreen Marking in Hong Kong in Three Subject Areas: A Rasch Measurement Perspective

  • Chapter
  • First Online:
Validating Technological Innovation

  • 302 Accesses

Abstract

During the series of validation studies investigating the implementation of onscreen marking in Hong Kong’s public examinations, an issue arose. The issue was that some markers were not as positive as might have been expected – given the English Language markers’ reactions to the enhanced support from the system. Concern was expressed about the feedback and support provided to markers on the accuracy of their marking. It was considered that this was an area that should be investigated with a view to possibly enhancing the amount and type of feedback provided to markers. This chapter therefore extends the investigation into OSM (see accounts of previous studies in this volume) into two areas: ease of use in the environment; and markers’ acceptance of OSM in the Hong Kong public examination context. In contrast to previous studies in this volume where there was a single subject area focus, this study took a heterogeneous approach. The sample forming the database for the investigation contained scripts from three subjects (English Language, Chinese Language, and Liberal Studies). Scripts comprised essays and short answer questions, as well as scripts written in either English or Chinese. Two scales assessing the ease of use and markers’ acceptance of OSM were investigated from a Rasch measurement perspective (a more sophisticated mode of measurement compared with classical test theory as described in Chap. 3 and below); with both scales showing good psychometric properties. The findings revealed that markers generally had a high level of perceived ease of use in the environment and the overall acceptance of OSM was positive. Differences of person measures across language, question type, and subject were compared and implications of the two scales for future validation research studies are briefly discussed.

The original version of this chapter was revised: The author names David Coniam, Peter Falvey, and Zi Yan were inserted. The erratum to this chapter is available at: http://dx.doi.org/10.1007/978-981-10-0434-6_15

This study originally appeared as: Yan, Zi. & Coniam, David (2013). Assessing the ease of use in the environment and markers’ acceptance of on screen marking: a Rasch measurement perspective. Educational Research and Evaluation, 19(5), 461–483. Reprinted with kind permission from Taylor & Francis.

An erratum to this chapter can be found at http://dx.doi.org/10.1007/978-981-10-0434-6_15

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Andrich, D. (2004). Controversy and the Rasch model: A characteristic of incompatible paradigms? Medical Care, 42(1), 1–7.

    Article  Google Scholar 

  • Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah: Erlbaum.

    Google Scholar 

  • Bos, W., Goy, M., Howie, S. J., Kupari, P., & Wendt, H. (2011). Editorial: Rasch measurement in educational contexts Special issue 2: Applications of Rasch measurement in large-scale assessments. Educational Research and Evaluation, 17, 413–417.

    Article  Google Scholar 

  • Chan, A. H. S., So, J. C. Y., & Tsang, S. N. H. (2011). Developing optimum interface design for on-screen Chinese proofreading tasks. In Proceedings of the 1st international conference on Human interface 2011. Held as Part of HCI International 2011, Orlando, FL, USA, 9–14 July 2011, Proceedings, Part II.

    Google Scholar 

  • Chen, C.-H., & Chien, Y.-H. (2005). Effect of dynamic display and speed of display movement on reading Chinese text presented on a small screen. Perceptual and Motor Skills, 100, 865–873.

    Article  Google Scholar 

  • Coniam, D. (2009a). A comparison of onscreen and paper-based marking in the Hong Kong public examination system. Educational Research and Evaluation, 15(3), 243–263.

    Article  Google Scholar 

  • Coniam, D. (2009b). Discrepancy essays: Natural phenomenon or problem to be solved? Melbourne Papers in Language Testing, 14(2), 1–31.

    Google Scholar 

  • Coniam, D. (2011). A qualitative examination of the attitudes of liberal studies markers towards onscreen marking. British Journal of Educational Technology, 42(6), 1042–1054.

    Article  Google Scholar 

  • Coniam, D. (2013). The increasing acceptance of onscreen marking – the ‘tablet computer’ effect. Journal of Educational Technology & Society, 16(3), 119–129.

    Google Scholar 

  • Coniam, D., & Yeung, A. (2010). Markers’ perceptions regarding the onscreen marking of liberal studies in the Hong Kong public examination system. Asia Pacific Journal of Education, 30(3), 249–271.

    Article  Google Scholar 

  • Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–339.

    Article  Google Scholar 

  • Fowles, D., & Adams, C. (2005, September). How does assessment differ when e-marking replaces paper-based marking? Paper presented at the 31st International Association for Educational Assessment Conference, Abuja, Nigeria.

    Google Scholar 

  • Hart, D. L., & Wright, B. D. (2002). Development of an index of physical functional health status in rehabilitation. Archives of Physical Medicine and Rehabilitation, 83, 655–665.

    Article  Google Scholar 

  • Johnson, M., Nádas, R., & Green, S. (2010). Marking essays on screen and on paper. Education Journal, 121, 39–41.

    Google Scholar 

  • Linacre, J. M. (2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3(1), 85–106.

    Google Scholar 

  • Linacre, J. M. (2006). A user’s guide to WINSTEPS/MINISTEP: Rasch-model computer programs. Chicago: Winsteps.com.

    Google Scholar 

  • Linacre, J. M. (2011). WINSTEPS: Rasch measurement computer program. Chicago: Winsteps.com.

    Google Scholar 

  • Muís, K. R., Winne, P. H., & Edwards, O. V. (2009). Modern psychometrics for assessing achievement goal orientation: A Rasch analysis. British Journal of Educational Psychology, 79, 547–576.

    Article  Google Scholar 

  • Powers, D., Kubota, M., Bentley, J., Farnum, M., Swartz, R., & Willard, A. (1997). A pilot test of on-line essay scoring. ETS Report RM-97-07.

    Google Scholar 

  • Rasch, G. (1960/1980). Probabilistic models for some intelligence and attainment tests. (Copenhagen, Danish Institute for Educational Research), expanded edition (1980) with foreword and afterword by B.D. Wright. Chicago: University of Chicago Press.

    Google Scholar 

  • Schulz, W., & Fraillon, J. (2011). The analysis of measurement equivalence in international studies.

    Google Scholar 

  • Shaw, S. (2008). Essay marking on-screen: Implications for assessment validity. E-Learning and Digital Media, 5(3), 256–274. Retrieved January 11, 2011, from http://www.wwwords.co.uk/elea/content/pdfs/5/issue5_3.asp.

    Google Scholar 

  • Shaw, S., & Imam, H. (2008, September). On-screen essay marking reliability: Towards an understanding of marker assessment behaviour. Paper presented at the IAEA 2008 conference, Cambridge, UK.

    Google Scholar 

  • Törmäkangas, K. (2011). Advantages of the Rasch measurement model in analysing educational tests: An applicator’s reflection. Educational Research and Evaluation, 17, 307–320.

    Article  Google Scholar 

  • Twing, J., Nichols, P., & Harrison, I. (2003). The comparability of paper-based and image-based marking of a high-stakes, large-scale writing assessment. International Association for Educational Assessment Conference, Manchester, 5–10 Oct 2003.

    Google Scholar 

  • Verhelst, N. D., & Glas, C. A. (1995). The one parameter logistic model. In G. H. Fischer & I. W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 215–237). New York: Springer.

    Chapter  Google Scholar 

  • Wang, W. C. (2000). Modeling effects of differential item functioning in polytomous items. Journal of Applied Measurement, 1, 63–82.

    Google Scholar 

  • Wendt, H., Bos, W., & Goy, M. (2011). On applications of Rasch models in international comparative large-scale assessments: A historical review. Educational Research and Evaluation, 17, 419–446.

    Article  Google Scholar 

  • Wright, B. D. (1992). IRT in the 1990s: Which models work best? Rasch Measurement Transactions, 6, 196–200.

    Google Scholar 

  • Wright, B. D. (1997). A history of social science measurement. Educational Measurement: Issues and Practice, 16(4), 33–45.

    Article  Google Scholar 

  • Yan, Z., & Bond, T. G. (2011). Developing a Rasch measurement physical fitness scale for Hong Kong primary school-aged students. Measurement in Physical Education and Exercise Science, 15, 182–203.

    Article  Google Scholar 

  • Yan, Z., & Coniam, D. (2013). Assessing the ease of use in the environment and markers’ acceptance of onscreen marking: A Rasch measurement perspective. Educational Research and Evaluation, 19(5), 461–483.

    Article  Google Scholar 

  • Yen, N.-S., Tsai, J.-L., Chen, P.-L., Lin, H.-Y., & Chen, A. L. P. (2011). Effects of typographic variables on eye-movement measures in reading Chinese from a screen. Behaviour & Information Technology, 30, 797–808.

    Article  Google Scholar 

  • Zhang, Y., Powers, D., Wright, W., & Morgan, R. (2003). Applying the online scoring network (OSN) to advanced program placement program (AP) Tests. ETS Research Report RR-03-12.

    Google Scholar 

  • Zumbo, B. D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4, 223–233.

    Article  Google Scholar 

  • Zwick, R., Thayer, D. T., & Lewis, C. (1999). An empirical Bayes approach to Mantel-Haenszel DIF. Retrieved on November 8, 2015, from: http://www.winsteps.com/winman/mantel_and_mantel-haenszel_dif.htm.

Download references

Author information

Authors and Affiliations

  1. Department of Curriculum and Instruction, The Education University of Hong Kong, Tai Po, Hong Kong

    David Coniam, Peter Falvey & Zi Yan

Authors
  1. David Coniam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Falvey
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zi Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Coniam .

Appendices

Appendix 1: 2012 HKALE/HKDSE: Onscreen Post-Marking Questionnaire

Now that you have marked, please fill in the questionnaire below – reflecting upon your experience of the onscreen marking process. All information collected is for research purposes only, will be kept in the strictest confidence, and will not be released to any other party.

figure a

Appendix 2: Finalised Instrument

Ease of Use in the OSM Environment Scale

  1. 01.

    How comfortable was the marking area in the assessment centre (general ambience, space, lighting, air-con etc.)?

  2. 02.

    How would you rate your computer proficiency?

  3. 03.

    Was the desktop at the right height for you?

  4. 04.

    How was screen resolution?

  5. 05.

    How comfortable were you reading off the screen?

  6. 06.

    How easily could you enlarge the screen image?

  7. 07.

    How easily could you scroll the screen image?

Acceptance of OSM Scale

  1. 01.

    How tired did your eyes get marking on screen?

  2. 02.

    How much pressure did you feel, knowing that your marking performance was being constantly monitored?

  3. 03.

    How helpful did you find the support and feedback from the OSM system?

  4. 04.

    Overall, how would you rate your onscreen marking experience?

  5. 05.

    How do you now feel about the move from paper-based to onscreen marking?

  6. 06.

    Would you prefer to mark on screen or on paper?

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer Science+Business Media Singapore

About this chapter

Cite this chapter

Coniam, D., Falvey, P., Yan, Z. (2016). The Sixth Validation Study: Assessing the Ease of Use in the Environment and Markers’ Acceptance of Onscreen Marking in Hong Kong in Three Subject Areas: A Rasch Measurement Perspective. In: Validating Technological Innovation. Springer, Singapore. https://doi.org/10.1007/978-981-10-0434-6_10

Download citation

  • DOI: https://doi.org/10.1007/978-981-10-0434-6_10

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-10-0432-2

  • Online ISBN: 978-981-10-0434-6

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation