Teachers as Testers: The Case of EOI Teachers in Galicia, Spain

Abstract

The network of EOI public language schools for adults, present in all of Spain with over 300 learning centers, provides a unique educational context within Europe in which teaching and testing are intertwined. These schools have a twofold function: they provide classes in foreign and second languages, which are in high demand, and they grant official language certificates in alignment with the Common European Framework of Reference (CEFR) by means of official exams that are open to non-students as well. In EOIs in Galicia, Spain, teachers of eleven different foreign languages not only teach, but are also intricately involved in the testing process, as they have extensive experience rating candidates’ oral and written performances and, as of 2012, also write tasks for the official certification exams. Extensive assessment training has been provided by the Galician government and been organized by teachers as well, as their interest in improving their assessment literacy has grown. These teachers’ experience in this process can shed light on teachers’ thoughts on their role in testing. A questionnaire (n = 114) and interviews (n = 6) were used to explore the impact EOI teachers in Galicia believe this training has had on their work. Findings reveal that, although their views of the training tend to stay within a narrow definition of assessment literacy, the teachers highly value this training as a way of not only improving their work as testers, but as a valuable tool for teaching as well.

Appendices

Appendices

1.1 Appendix 1

1.1.1 Questionnaire About the Assessment Training of EOI Teachers in Galicia

I would like to ask you to participate in this questionnaire about the training in assessment of teachers that work in EOIs in Galicia. This research will form part of my MA Dissertation for Lancaster University, and its aim is to study the training needs in assessment of EOI teachers and the impact that this training has on the work of teachers. This study will hopefully help to disseminate the important role that EOIs have in language assessment.

Your participation will allow teachers, schools and the administration to know your specific training needs in assessment, which can lead to training that is adapted to our context and designed to have the best possible impact on our work.

Your responses are absolutely confidential and anonymous. If you wish to receive information about the results of this study and you wouldn’t mind being contacted to participate in an interview, you can choose to leave your contact details at the end of the survey. If you have any questions, please contact me.

• 1. Do you or have you worked in an EOI in Galicia?

  • Yes

  • No

• 2. How many times have you participated in a standardization session of the rating criteria with teachers from your department or other departments?

  • Never (Go on to question 5)

  • 1–2 days

  • 3–5 days

  • 6–10 days

  • More than 10 days

• 3. Mark from 0 to 3 the impact that these standardization sessions have had on the following aspects of your work: (If you have not participated in an activity, please leave this line blank.)

	0	1	2	3
	No impact	Small impact	Some impact	Large impact
Creating speaking tasks
Creating writing tasks
Selecting texts to use in exams
Assessing the spoken performance of candidates in exams
Assessing the written performance of candidates in exams
Creating tasks for use in the classroom
General teaching practice

• 4. Would you like to add anything about the impact of standardization sessions?

1.1.2 Training in Assessment

• 5. Have you received any type of training in assessment? (Training in assessment defined here as formal instruction in a course, workshop, etc. on the theory or practice of assessment and/or familiarization with the CEFR.)

  • Yes

  • No (Go on to question 12)

• 6. Number of days in which you participated in an assessment training event:

  • 1–2 days

  • 3–5 days

  • 6–10 days

  • More than 10 days

• 7. In what period did this training occur? (Mark all of the correct options.)

  • In the last 1–3 years

  • In the last 3–8 years

  • More than 8 years ago

• 8. More information about these training events: (Please detail briefly where these training events occurred and who they were organized by.)

• 9. Which of the following aspects were dealt with in this training? (Mark all of the options that were part of this training.)

  • Creating tasks for reading/listening tests (receptive skills)

  • Creating tasks for writing/speaking tests (productive skills)

  • Selecting texts for use in exams

  • Theory of the skills assessed (reading, writing, listening, and speaking)

  • Editing audio with Audacity or similar

  • Principles of assessment (validity, reliability, etc.)

  • Familiarization with the CEFR

  • Familiarization with the certification exam test format

  • Use of rating scales

  • Other: ___________

• 10. Mark from 0 to 3 the impact that this training in assessment has had on the following aspects of your work: (If you have not participated in an activity, please leave this line blank.)

	0	1	2	3
	No impact	Small impact	Some impact	Large impact
Creating tasks for reading/listening tests (receptive skills)
Creating tasks for writing/speaking tests (productive skills)
Selecting texts to use in exams
Editing audio with Audacity
Assessing the spoken performance of candidates in exams
Assessing the written performance of candidates in exams
Creating tasks for use in the classroom
General teaching practice

• 11. Would you like to add anything about the impact of this training in assessment?

1.1.3 Training Needs in Assessment

• 12. In which of the following aspects do you think you need more training? (Mark all of the options you consider you need more training in.)

  • Creating tasks for reading/listening tests (receptive skills)

  • Creating tasks for writing/speaking tests (productive skills)

  • Selecting texts for use in exams

  • Theory of the skills assessed (reading, writing, listening, and speaking)

  • Editing audio with Audacity

  • Principles of assessment (validity, reliability, etc.)

  • Familiarization with the CEFR

  • Standardization of the rating criteria with speaking samples

  • Standardization of the rating criteria with writing samples

  • Familiarization with the certification exam test format

  • Use of rating scales

  • Other: ___________

• 13. How much time would you be willing to invest in training in assessment?

  • 3–4 h

  • 5–10 h

  • 11–20 h

  • 20–30 h

  • Other: ___________

• 14. Which would be the most ideal format for this training?

  • Online course with a tutor

  • Online materials for self-study

  • Face-to-face course

  • Combination of online and face to face course

  • Other: ___________

• 15. Would you like to add anything about your training needs in assessment?

1.1.4 Other Information

• 16. What EOI do you currently work at or have you last worked at?

• 17. Do you work in the main EOI or in an EOI section?

  • Main EOI

  • EOI section in the same city as the EOI

  • EOI section in a town different from main EOI

• 18. What language(s) do you teach?

• 19. How long have you taught at an EOI?

• 20. How long have you taught in a secondary school?

• 21. How long have you taught in a different sector?

• 22. Which of the following activities have you done as part of your work at an EOI? (Mark all of the correct options.)

  • Creating tasks for certification exams

  • Creating tasks for other exams

  • Administering final or certification exams

  • Rating the spoken performance of candidates on final or certification exams

  • Rating the written performance of candidates on final or certification exams

• 23. Level of studies:

• 24. Gender:

• 25. Age:

• 26. If you wish to leave your contact information you can do so. You will receive further information about the results of this study and you may be contacted to participate in a follow-up interview. Your personal information will remain anonymous.

Thank you for your participation!

1.2 Appendix 2

Areas in which the respondents feel they need more training

Area	Valid percent
Creating tasks for listening/reading tests (receptive skills)	60.2% (n = 68)
Creating tasks for speaking/writing tests (productive skills)	51.3% (n = 58)
Selecting texts for use in tests	61.1% (n = 69)
Theory of the four skills	40.7% (n = 46)
Editing audio with Audacity	40.7% (n = 46)
Principles of assessment (validity, reliability, etc.)	54.9% (n = 62)
Familiarization with the CEFR	32.7% (n = 37)
Standardization of the rating criteria with speaking samples	69.9% (n = 79)
Standardization of the rating criteria with writing samples	69.0% (n = 78)
Familiarization with the format of certification exams	17.7% (n = 20)
Use of rating scales	51.3% (n = 58)

1.3 Appendix 3

1.3.1 Interview Questions

• 1. As you know, the topic of this study is to understand the assessment literacy and needs of EOI teachers. Please describe the training you have received in assessment.

• 2. What impact has this training had on your work? (Preparing tests, in class/teaching…?) What about the impact of standardization sessions?

• 3. What further training in assessment do you feel you need? And your colleagues?

• 4. What difference do you think there is between having a team of test-writers and the current system in which all teachers write tasks for tests?

• 5. “Assessment literacy” is defined as the combination of knowledge and practical skills of testing, along with an understanding of the principles of assessment and its role. Do you think that EOI teachers ought to have a deep understanding of assessment?

• 6. What do you think the role of EOIs should be in language certification?

• 7. Would you like to add anything else?

1.4 Appendix 4

Interview respondent profiles, organized by degree of training in assessment

Respondent	Specialization	Experience teaching in EOI	Other teaching experience	Training in assessment
Respondent 1	Secondary school	2 years	15–20 years	None
Respondent 2	EOI	5–10 years	6–12 months	3–5 days
Respondent 3	EOI	10–15 years	None	6–10 days of courses
Respondent 4	EOI	5–10 years	None	6–10 days of courses
				Self-study by reading
Respondent 5	EOI	+20 years	10–15 years	Extensive training throughout career
Respondent 6	EOI	5–10 years	5–10 years	+100 h training
				Courses in Spain and abroad
				Self-study by reading
				Teacher trainer