Testing Usefulness of Reading Comprehension Exams Among First Year Students of English at the Tertiary Level in Tunisia

Abstract

This study investigated testing usefulness (Bachman & Palmer, 1996) of reading comprehension among first year students of English at the tertiary level in an EFL context. T Data were gathered by means of two questionnaires and a corpus which included 173 first year students’ graded reading exams. A questionnaire was administered to 64 students and 21 teachers which was collected at different institutes and universities. Scores of the reading exams were meant to check the appropriateness of the construct and rater consistency. Questionnaire results indicated that most of the reading comprehension examinations had an acceptable level of reliability and a moderate level of authenticity and interactiveness. It was also revealed that these exams had a high level of construct validity and practicality. In addition, the findings indicated that the achievement reading tests had a harmful impact on first year students, whereas they had a beneficial washback effect on the teachers. On the other hand, the results of the reading scores’ analysis using Cronbach alpha to estimate the reading tests’ reliability showed that there were only three tests among six which had an acceptable reliability of (∝=.70). Concerning the findings of the construct validity assessment procedure of the reading exams, they showed that only the reading part in comprehension, composition, grammar test 1 and the reading section in comprehension, composition, grammar test 2 were proven to be construct valid. These results contradicted those of the questionnaires. Actually, the participants did not provide trustworthy answers in the reliability and construct validity parts. Therefore, most of the reading midterm and final exams designed by the teachers at the tertiary level in Tunisia had a low reliability and construct validity, a moderate authenticity and interactiveness. Nevertheless, the reading exams had a high practicality and a beneficial impact on the teachers, but had a harmful impact on first year learners. The reading exams had a low, since reliability and construct validity which are two important criteria were proven to be threatened. The study had implications for test design.

Appendices

Appendix 1

Students’ Questionnaire

This questionnaire investigates the problems in reading comprehension tests among first year English students at the tertiary level in Tunisia and tries to find different solutions to them. This questionnaire may take you about 15 min to fill out. The information you provide will be strictly confidential and used for academic purposes only. Your cooperation is very important for this research project. Please, answer truthfully.

Section A—Background Information

• 1. Age: ${\displaystyle \begin{array}{l}18\hbox{-} 22\\ {}\mathrm{More}\ \mathrm{than}\ 22\end{array}}$

• 2. Rank the skills below from the most important (1) to the least important (4)

$$ {\displaystyle \begin{array}{cc}\mathrm{Reading}& \mathrm{Speaking}\\ {}\mathrm{Writing}& \mathrm{Listening}\end{array}} $$

• 3. What is your level in the reading skill?

$$ {\displaystyle \begin{array}{ll}\mathrm{Poor}& \mathrm{Good}\\ {}\mathrm{Average}& \mathrm{Excellent}\end{array}} $$

Section B—Test usefulness

• Part 1: Reliability

• 1. Did it happen to you that you got an undesirable mark in the reading test/exam during this year?

	Yes	No
If Yes, this occurred because of:
	Non-understanding of the text	Bad emotional state
	No world knowledge for the text	Temporary illness
	Length of the text	Others (please specify)
	The topic of the text was boring

• 2. Which testing techniques did your reading tests/exams include this year?

Multiple-choice questions	True/False questions	Gap-filling tasks
Information transfer	Paraphrasing	Summarising
Guessing the meaning of unfamiliar words from context

• Identifying referents [e.g., What does the underlined word ‘it’ (line 25) refer to?]

• Short-answer questions (The answers to these questions may vary from a word or phrase to one or two sentences)

• Others (please specify)………………………………

• 3. How often did your teacher give you a detailed scoring key in the reading test during this year?

$$ {\displaystyle \begin{array}{ccc}\mathrm{Never}\ \left(0\%\right)& \mathrm{Sometimes}\ \left(50\%\right)& \mathrm{Always}\ \left(100\%\right)\\ {}\mathrm{Rarely}& \mathrm{Often}& \end{array}} $$

• 4. State the extent to which you agree or disagree with the two following statements [Strongly Disagree (SD); Disagree (D); Undecided (U); Agree (A); Strongly Agree (SA)]

N°	Statements	SD	D	U	A	SA
1	At least, two different teachers should correct the reading test and exam paper.
2	In the reading tests, you should become identified by number not name, like in the final exam.

• 5. How often did you sit this year for a reading test/ exam in distracting conditions (e.g., unquiet class, bad conditions of desks and chairs,)?

$$ \mathrm{Never}\ \left(0\%\right)\kern1em \mathrm{Sometimes}\ \left(50\%\right)\kern1em \mathrm{Always}\ \left(100\%\right)\kern1em \mathrm{Rarely}\kern1em \mathrm{Often} $$

• 6. How do you think of the reading tests/exams that you took this year? They are…

$$ \mathrm{Long}\kern2.50em \mathrm{Neither}\ \mathrm{long}\ \mathrm{nor}\ \mathrm{short}\kern2.50em \mathrm{Short} $$

• 7. Did your teacher permit you to choose between tasks in the reading test/exam?

  during this year?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• 8. What do you think of the reading comprehension test/exam tasks that you took this year? They are…

$$ \mathrm{Ambiguous}\kern5.50em \mathrm{Unambiguous}/\mathrm{clear} $$

• 9. What do you think of the reading test/exam questions of this year? They are…

$$ \mathrm{Clear}\kern3.50em \mathrm{Unclear}\kern3.50em \mathrm{Guided}\kern3.50em \mathrm{Vague} $$

• 10. What do you think of the reading tests/exams that you took this year? They are…

$$ {\displaystyle \begin{array}{cccc}1.& \mathrm{Well}\hbox{-} \mathrm{presented}:& \kern2em \mathrm{Well}\hbox{-} \mathrm{typed}& \kern2em \mathrm{well}\hbox{-} \mathrm{reproduced}\\ {}2.& \mathrm{Badly}\hbox{-} \mathrm{presented}:& \kern2.5em \mathrm{Badly}\hbox{-} \mathrm{typed}& \kern3em \mathrm{poorly}\hbox{-} \mathrm{reproduced}\end{array}} $$

• 11. How many times did your teacher inform you this year about the aspects of the reading test/exam [format and testing techniques (some of them are mentioned in page 2)] before you took it?

$$ {\displaystyle \begin{array}{ccc}0\ \mathrm{time}& 2\ \mathrm{time}\mathrm{s}& 4\ \mathrm{time}\mathrm{s}\\ {}1\ \mathrm{time}& 3\ \mathrm{time}\mathrm{s}& \end{array}} $$

• Part 2: Construct validity

• 1. Do the reading tests/exams that you took this year test only your ability to understand a text?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• If No, explain……………………………………………………………...

• 2. What do you think of most of the reading test/exam questions? They are…

• Difficult to understand

• Neither difficult nor easy to understand

• Easy to understand

• 3. Did all your class often take the same reading test/exam on the day of the examination during this year?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• Part 3: Authenticity

• 1. Please, tick (√) one box in each row

N°	Questions	N	R	S	O	A
1	How often did the reading test/exam tasks that you took this year represent situations, which are similar to what you will encounter in your life?
2	How often did the text in the reading test/exam have a topic, which is real and not imaginary?

• 2. How did most of the vocabulary task(s) in the reading tests/exams that you took this year represent?

  • 1. They represented question(s) about the meanings of words or expressions

  • 2. They represented paragraph(s) to fill them in with given words

  • 3. They represented question(s) about the meanings of words or expression and paragraph(s) to fill them in with given words

3. How were the topics in the reading test/exam tasks? They were …

$$ {\displaystyle \begin{array}{cc}\mathrm{Interesting}& \mathrm{Humorous}\\ {}\mathrm{Enjoyable}& \mathrm{Boring}\end{array}} $$

• Part 4: Interactiveness

• 1. How often did responding to the reading test/exam tasks require your topical knowledge?

$$ {\displaystyle \begin{array}{ccc}\mathrm{Never}\ \left(0\%\right)& \mathrm{Sometimes}\ \left(50\%\right)& \mathrm{Always}\ \left(100\%\right)\\ {}\mathrm{Rarely}\hfill & \mathrm{Often}\hfill & \end{array}} $$

• 2. What did the understanding often required in the reading test/exam tasks involve? It involves...

  • 1. A large knowledge of the English language

  • 2. Neither large nor small knowledge of the English language

  • 3. A small knowledge of the English language

3. Please, tick (√) one box in each row

N°	Questions	N	R	S	O	A
1	How often did it happen that the reading test/exam tasks depend on each other?
2	How often did the reading test/exam tasks cause you any emotional threat?

• Part 5: Impact

• 1. Have you often prepared for the reading comprehension test/sub-test and exam?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• If No, why?...................................................................................................

• ............................................................................................................................

• 2. Did your teacher always inform you this year about the aspects of the reading test/exam [format and testing techniques (some of them are mentioned in page 2)] before you took it?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• 3. Did your teacher often ask you to suggest appropriate tasks for use in a reading test/exam?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• 4. Did your teacher often ask you to suggest appropriate marking criteria for use in a reading test/exam?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• 5. How many times did you receive remarks about your strengths and weaknesses in the two reading tests that you took this year?

$$ {\displaystyle \begin{array}{cc}0\ \mathrm{time}& 2\ \mathrm{time}\mathrm{s}\\ {}1\ \mathrm{time}& \end{array}} $$

• Part 6: Practicality

• 1. Please, tick (√) one box in each row

	Questions	N	R	S	O	A
1	How often did you complete the reading test/exam within the fixed time?
2	How often have the reading tests/exams been managed without problems?

Thank you for your cooperation

Appendix 2

Teachers’ Questionnaire

This questionnaire investigates problems in reading comprehension tests among first year English students at the tertiary level in Tunisia and tries to find different solutions to them. This questionnaire may take you about 15 min to fill in. The information you provide will be strictly confidential and used for academic purposes only. Your cooperation is very important for this research project. Please, answer truthfully.

Section A—Background Information

• 1. Name (optional): ...................................................................................................

• 2. Academic status: Please, tick (√) the appropriate one.

$$ {\displaystyle \begin{array}{cc}\mathrm{Professor}\hfill & \mathrm{Assistant}\ \mathrm{professor}\hfill \\ {}\mathrm{Secondary}\ \mathrm{school}\ \mathrm{teacher}\hfill & \mathrm{Other}\ \left(\mathrm{please}\ \mathrm{specify}\right)\dots \dots \dots \\ {}\mathrm{Assistant}\hfill & \end{array}} $$

• 3. Teaching first year reading course:

$$ {\displaystyle \begin{array}{cc}\mathrm{Less}\ \mathrm{than}\ 6\ \mathrm{years}& 11\hbox{--} 15\ \mathrm{years}\hfill \\ {}\hfill 6\hbox{--} 10\ \mathrm{years}\hfill & \hfill \mathrm{More}\ \mathrm{than}\ 15\ \mathrm{years}\hfill \end{array}} $$

• 4. Training:

• Have you received any type of training in testing reading?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• Please explain whether you answer yes or no .................................................. ................................................................................................................................

Section B—Test usefulness

• Part 1: Reliability

• 1. Which reading testing technique(s) do you often use in your reading tests/exams?

$$ {\displaystyle \begin{array}{cc}\mathrm{Multiple}\hbox{-} \mathrm{choice}\ \mathrm{questions}& \mathrm{True}/\mathrm{False}\ \mathrm{questions}\\ {}\mathrm{Short}\hbox{-} \mathrm{answer}\ \mathrm{questions}& \mathrm{Gap}\hbox{-} \mathrm{filling}\ \mathrm{tasks}\\ {}\mathrm{Information}\ \mathrm{transfer}& \mathrm{Paraphrasing}\\ {}\mathrm{Summarizing}& \mathrm{Identifying}\ \mathrm{referents}\\ {}\mathrm{Guessing}\ \mathrm{the}\ \mathrm{meaning}\ \mathrm{of}\ \mathrm{unfamiliar}\ \mathrm{words}\ \mathrm{from}\ \mathrm{context}& \\ {}\mathrm{Others}\ \left(\mathrm{please}\ \mathrm{specify}\right)\dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots .& \hfill \hfill \end{array}} $$

• 2. Do you often provide a detailed scoring key for the first year reading test/exam?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• 3. Have you received any type of training in scoring the first year reading test/exam?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• 4. Do you often precise the acceptable responses at the outset of scoring?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• 5. State the extent to which you agree or disagree with the two following statements [Strongly Disagree (SD); Disagree (D); Undecided (U); Agree (A); Strongly Agree (SA)]

	Statements	SD	D	U	A	SA
1	Each copy of a reading test and final exam should be scored by at least two independent scorers
2	In the reading tests, first year students should become identified by number not name, like in the final exam

• 6. How are your first year reading comprehension tests/exams? They are…

$$ \mathrm{Long}\kern0.5em \mathrm{short}\kern0.5em \mathrm{Neither}\ \mathrm{long}\kern0.20em \mathrm{nor}\kern0.20em \mathrm{short} $$

• 7. Do you often allow your first year students to choose between tasks in a reading examination?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• 8. How often do you inform your first-year students about the reading test/exam format before finalizing it?

$$ {\displaystyle \begin{array}{ccc}\mathrm{Never}\hfill & \mathrm{Sometimes}& \mathrm{Always}\\ {}\mathrm{Rarely}& \mathrm{often}\hfill & \end{array}} $$

• Part 2: Construct validity

• 1. Do your first year reading test tasks only measure your students’ ability to understand a text?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• If No, explain..........................................................................................................

• 2. Do you often take into consideration a given level of a students’ category in designing the reading exam questions?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• If Yes, what level of students do you often take in mind?

• The highest proficient students

• The medium proficient students

• The least proficient students

• 3. Do you often give different varieties of examinations to the first-year students on the day of the reading test?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• Part 3: Authenticity

• 1. Please, tick (√) one box in each row

	Questions	N	R	S	O	A
1	How often do you opt for real-world first year reading exam tasks?
2	How often do you give your first-year students a text which has a real-world topic in the reading exam?

• 2. What do the vocabulary task(s) that you often design in the first year reading tests represent?

  • 1. They represent question(s) about the meanings of words or expressions

  • 2. They represent paragraph(s) to be filled in with given words

  • 3. They represented questions about the meanings of words or expression and paragraphs to be filled in with given words

• Part 4: Interactiveness

• 1. Do your reading exams always contain some tasks which involve your first-year students’ topical knowledge?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• 2. Do you often take into consideration first year students’ characteristics (e.gs., age, nationality, educational background) in designing the reading exam tasks?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• If No, why?

• ...............................................................................................................................

What does the understanding often required in your reading test tasks involve? It involves…

• 1. A wide range of first year students’ areas of language knowledge

• 2. Neither wide nor narrow range of first year students’ areas of language knowledge

• 3. A narrow range of first year students’ areas of language knowledge

• 3. How often do you design dependent first year reading comprehension test tasks?

$$ {\displaystyle \begin{array}{ccc}\mathrm{Never}\hfill & \mathrm{Sometimes}\hfill & \mathrm{Always}\hfill \\ {}\mathrm{Rarely}& \mathrm{Often}& \end{array}} $$

• Part 5: Impact

• 1. What do you often assess in your first year reading comprehension test?

Students’ ability to understand a text Students’ other abilities (please specify)...................................... Students’ ability to write

• 2. Which type of testing do you often use in the first year reading test?

  • 1. Direct testing: A testing method that closely matches the reading comprehension ability being measured

  • 2. Indirect testing: A testing method that measures abilities related to the reading comprehension ability being tested, rather than this ability itself

  • 3. Direct and indirect testing

• 3. What is your reading test/exam based on?

$$ {\displaystyle \begin{array}{ll}\mathrm{Course}\ \mathrm{objectives}& \mathrm{what}\ \mathrm{is}\ \mathrm{taught}\ \mathrm{in}\ \mathrm{the}\ \mathrm{classroom}\\ {}\mathrm{Course}\ \mathrm{content}& \mathrm{Other}\ \left(\mathrm{please}\ \mathrm{specify}\right)\dots \dots \dots \dots \dots \dots \end{array}} $$

• 4. Do you always familiarise your first-year students with the reading test/exam

  format before they take it?

$$ \mathrm{Yes}\kern15.32em \mathrm{No} $$

• Part 6: Practicality

• 1. Please, tick (√) one box in each row

N°	Questions	N	R	S	O	A
1	How often are administrative details established clearly before the reading test/exam?
2	How often do your first-year students complete the reading test/exam within the set time frame?
3	How often is your scoring system for the reading test/ exam feasible in your time frame?
4	How often have the material resources (space, equipment, and materials) been provided to the reading test/exam’s design?
5	How often have the material resources been provided to the reading test/exam’s administration?
6	How often have the reading tests/exams been administered smoothly (without problems)?

Thank you for your cooperation