Research Methodologies for Exploring the Validity of a Test

  • Cyril J. Weir
Part of the Research and Practice in Applied Linguistics book series (RPAL)


There are many excellent general textbooks available on doing research and you will find references at the end of this chapter. To find out more about qualitative and quantitative research design you are referred to them, and references are also given on specific methodologies such as questionnaire design, interview, and verbal protocols, statistical analysis procedures, and discourse analyses of testee performance.


Research Methodology Task Difficulty Verbal Report Test Taker Cognitive Complexity 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Further reading

  1. Brown, Anderson, Shillcock and Yule (1984) Google Scholar
  2. Foster and Skehan (1996, 1999) Google Scholar
  3. Mehnert (1998) Google Scholar
  4. Norris et al. (1998)Google Scholar
  5. Ortega (1999) Google Scholar
  6. Skehan (1996) Google Scholar
  7. Skehan and Foster (1997) Google Scholar
  8. Wigglesworth (1997) Google Scholar
  9. Lazaraton (2002), O’Loughlin, (2001), Pavlou (1997) and Shohamy (1994) analysed discourse features across different speaking tasks and detail useful methods for analysing such data.Google Scholar
  10. Cumming (1997) provides a very useful summary of the literature that has looked at the characteristics of written texts in language tests.Google Scholar
  11. Scott (2002) Wordsmith Tools website: /wordsmith/version3/index.html
  12. Urquhart and Weir (1998: 270–95) provide a full account of a range of procedures and methods of analysis, with examples which may be used in studies which attempt to unpack the reading process operationalized through a reading test.Google Scholar
  13. On what happens in the test taking process or for further ideas on how to investigate this for yourself, it is worth looking at:Google Scholar
  14. Buck (1991), Hale and Courtney (1994) and Wu Yian (1998) on listening.Google Scholar
  15. Anderson et al (1991), Allan (1992), Crain-Thoreson et al (1997) Nevo (1989), Perkins (1992), Phakiti (2003), Storey (1995 and 1997) and Weir et al (2000) on readingGoogle Scholar
  16. Ross (1992) on speakingGoogle Scholar
  17. Smagorinsky (1994) on writingGoogle Scholar
  18. On protocol analysis Google Scholar
  19. Cohen (1984, 1988, 1994 and 1997) on verbal reports for investigating test-takingGoogle Scholar
  20. Gass and Mackey (2000) for a useful theoretical and practical account of verbal protocol analysisGoogle Scholar
  21. Green (1998) on verbal protocol analysis in language testing research.Google Scholar
  22. Ericsson and Simon (1993) on protocol analysis: verbal reportGoogle Scholar
  23. Pressley and Afflerbach (1995) on verbal protocols for readingGoogle Scholar
  24. Stratman and Hamp-Lyons (1994) on concurrent think-aloud protocols.Google Scholar
  25. Baker (1997) is one of the few accessible accounts of using IRT analysis.Google Scholar
  26. Bryman and Cramer (2001) is an accessible guide to using SPSS.Google Scholar
  27. Crocker and Algina (1986) provide a solid and accessible account of the statistics you may need.Google Scholar
  28. Fulcher (2003) Chapter 7 provides an accessible account of correlation, factor analysis, multi-trait, multi-method studies and generalizability studies.Google Scholar
  29. Kim and Muller (1978a) There are no easy introductions to factor analysis but this is as close as it gets at the moment.Google Scholar
  30. Kim and Muller (1978b) is the next step up on the previous reference.Google Scholar
  31. McNamara (1996: Chapters 5–8 in particular) is one of the few readable and comprehensive accounts of using Rasch analysis, with examples of how to interpret the output.Google Scholar
  32. Myford and Wolfe (2000, 2003, 2004) for accessible accounts of MFR.Google Scholar
  33. Norris et al. (1998) for an extended discussion on how to estimate task difficulty along a number of parameters and a suggested procedure for operationalizing this SPSS Inc. (2002) The handbook that goes with the program.Google Scholar
  34. O’Sullivan, Weir and Sa ville (2002). Describes the rationale and development of the observation checklists described above.Google Scholar
  35. Weir and Milanovic (2003) provide a detailed coverage of what one Exam Board, Cambridge ESOL, does in this area. One of the few open and frank accounts of what goes on behind the scenes at one of the world’s leading test deliverers.Google Scholar
  36. Yang and Weir (1998) for a discussion of how parallel forms are achieved for the College English Test (CET) in China taken by over 10 million candidates a year.Google Scholar

Washback studies

  1. Green (2003) provides a very good survey of the literature in the field, some useful data collection instruments, and some innovative methods of analyses.Google Scholar
  2. Cheng (2004) provides a recent comprehensive study of the effect of the introduction of a new examination with a useful methodology section.Google Scholar
  3. Cheng and Watanabe (eds.) (2004) provide a useful and varied set of papers on recent developments in this areaGoogle Scholar
  4. Wall (2004) provides an interesting account of washback in the Sri Lankan context and sets it in the context of change theoryGoogle Scholar

Recommended reading

  1. Allan (1995) on questionnaires.Google Scholar
  2. Allwright (1988) on observation.Google Scholar
  3. Banerjee and Luoma (1997) on qualitative approaches to test validation.Google Scholar
  4. Brown (1991) on research methods.Google Scholar
  5. Cohen, Manion and Morrison (2000) on research methods in education.Google Scholar
  6. Denicolo and Pope (eds.) (1997) on interviewing.Google Scholar
  7. Foddy (1994) on constructing questions for interviews and questionnaires: language research.Google Scholar
  8. Fulcher (2003: chapter 7) provides an accessible overview of correlation, factor analysis, multi-trait, multi-method studies, generalizability studies, multi-faceted Rasch analysis, expert judgement, questionnaires and interviews, discourse analysis and verbal protocol analysisGoogle Scholar
  9. Hatch and Lazaraton (1997) on design and statistics for applied linguistics.Google Scholar
  10. Miles and Huberman (1994) on qualitative data analysis.Google Scholar
  11. Oppenheim (1992) on questionnaire design, interviewing and attitude measurement.Google Scholar
  12. Patton (2002) on qualitative research methods.Google Scholar
  13. Scwartz and Sudman (eds.) (1996) on the methodology for determining cognitive and communicative processes in survey researchGoogle Scholar
  14. Urquhart and Weir (1998: chapter 5) provide detailed accounts of how to do research in the area of reading and provide examples of instruments and analysis.Google Scholar
  15. Weir and Roberts (1994) describe the advantages and disadvantages of a variety of survey instruments and how to construct and operationalize themGoogle Scholar
  16. Weir, Yang and Jin (2000) investigate the construct of reading by a variety of methods.Google Scholar

Copyright information

© Cyril J. Weir 2005

Authors and Affiliations

  • Cyril J. Weir
    • 1
  1. 1.Centre for Research in Testing, Evaluation and Curriculum (CRTEC)Roehampton UniversityUK

Personalised recommendations