Accuracy in identifying students’ miscues during oral reading: a taxonomy of scorers’ mismarkings

Abstract

Informal reading inventories (IRI) and curriculum-based measures of reading (CBM-R) have continued importance in instructional planning, but raters have exhibited difficulty in accurately identifying students’ miscues. To identify and tabulate scorers’ mismarkings, this study employed examiners and raters who scored 15,051 words from 108 passage readings by students in Grades 5 and 6. Word-by-word scoring from these individuals was compared with a consensus score obtained from the first author and two graduate students after repeated replaying of the audio from the passage readings. Microanalysis conducted on all discrepancies identified a cumulative total of 929 mismarkings (range = 1–37 per passage) that we categorized in 37 unique types. Examiners scoring live made significantly more mismarkings than raters scoring audio recordings, t(214) = 4.35, p = .0001, with an effect size of d = 0.59. In 98% of the passages, scorers disagreed on the number of words read correctly—the score used for screening and progress monitoring decisions. Results suggest that IRIs and CBM-Rs may not be accurate as diagnostic tools for determining students’ particular word-level difficulties.

Appendix

Description of mismarking types

	Mismarking type	Description of mismarking
1	Student restarts	The student begins reading but then stops to read the title or asks the examiner if he/she should read the title before restarting the passage at the beginning. Some examiners inappropriately attempt to stop and restart the timer (rather than treating this as a repetition), thus contributing to misidentifying the TWR
2	Read title	When the student reads the title before reading the passage, some examiners start the timer at the title and some start at the first word of the passage. If the title and the first words of the passage are the same, it is unclear which the student is reading until additional words are read. The confusion over the title contributes to misidentifying the TWR
3	Repeated words around SC	The student makes a mistake in a sentence with words that are used multiple times. It is not always clear to which of the repeated words the student returns when self-correcting. This can create multiple possibilities for how the reading should be scored or can cause the examiner to add or miss mistakes
4	Rare word	The passage contains a low frequency word or word with multiple pronunciations, and the examiner does not know the correct pronunciation or all acceptable pronunciations. Therefore, the student’s reading is incorrectly scored
5	Hyphenated word	The hyphen rule for scoring is to count the hyphenated word as one if the parts cannot stand alone (e.g., t-shirt), but as separate words for scoring if each part can stand alone (e.g., mother-in-law). Mismarkings can occur if the rater counts multiple parts of the hyphenation as incorrect when the student makes a mistake on only one part
6	Last words	The examiner may add or omit marks due to preoccupation with the timer at the end of the 1-min, the chime obscuring the student’s reading, and the students speeding up around the 1-min mark. These issues make it difficult to accurately record the last word said within the time limit
7	Unclear when timer stops	An examiner may not start the timer, may not use a timer that chimes at 1 min, may allow the student to continue reading after the minute time limit, or may experience a timer malfunction—all of which result in uncertainty regarding the correct stopping point
8	Examiner stops early	The examiner stops the student’s reading at any point before 1 min has lapsed. Because the examiner stopped the student early, the TWR would have to be manually calculated (but was not)
9	Miscue on SC	The student re-reads a word or portion of the passage in an attempt to correct a perceived mistake, but still reads the word(s) incorrectly. The examiner does not mark the mistake on the self-correct
10	Miscue then SC	The student re-reads a word or portion of the passage and corrects a mistake, but the examiner still marks it as incorrect
11	Student clarifying word after reading	The student reads a word correctly but then asks the examiner to confirm that was correct. The examiner marks the correctly read word as incorrect
12	Examiner gives the word	The student asks the examiner to say a word or has hesitated for 3-s, thus prompting the examiner to tell the student the word. The examiner does not mark this word as incorrect
13	Dialect/colloquialism	The student is a native English speaker but uses a regional dialect. The examiner mistakenly marks words pronounced in the regional dialect as incorrect
14	Accent	The student is not a native English speaker and pronounces words in a way that is influenced by the student’s first language. The examiner mistakenly marks these as incorrect (either due to lack of knowledge or lack of consistency in applying the scoring procedure). [Additional note: The word says is often pronounced with a long/ā/sound by students whose first language is Spanish. Because says is a high frequency sight word, this pronunciation was considered incorrect. When examiners did not mark it as incorrect, it was counted as an omit mismarking.]
15	Stutter	Student stutters on the first letter(s) of a word but otherwise reads the word correctly
16	Drawing out pronunciation	The student draws out a sound within a word. The examiner either mistakenly marks this as incorrect if the student ultimately reads the word correctly or does not mark as incorrect the word that ultimately was mispronounced
17	Pause during word	Student pauses after starting to say a word, not making any sound as in a hesitation, but resumes reading the word correctly within the 3 s limit. The examiner mistakenly marks these as incorrect
18	Unclear pronunciation	The student does not enunciate the word, so the examiner marks what is perceived incorrectly (either adding or omitting mistakes)
19	Words read quickly	The student reads very quickly and may seem to be reading multiple words as though they were only one word (i.e., unclear where one word ended and the other began). The examiner mistakenly marks these as incorrect
20	Adding—s ending	The student incorrectly adds an -s to the end of a word, and the examiner does not mark it as incorrect
21	Dropping—s ending	The student reads a word without saying the -s at the end of the word, but the examiner does not mark it as incorrect
22	Dropping—ed ending	The student reads the word without the -ed ending, but the examiner does not mark it as incorrect
23	Derivation of word	The student says another form of the word in the passage, but the examiner does not mark it as incorrect
24	Formed contraction	The student reads two words in their contracted form, and the examiner does not mark it as incorrect
25	Replaced with other word	The student reads a word other than the word in the passage, and the examiner does not mark it as incorrect
26	Letter reversal	The student reverses letters within a word when reading it, but the examiner does not mark it as incorrect
27	Replaced syllable	Student reads a word with a change at the syllable level, but the examiner does not mark it as an incorrect
28	Add phoneme	The student adds a sound when reading a word, but the examiner does not mark it as incorrect. Note: This does not include adding -s to the end of a word, which is considered a separate category
29	Left out phoneme/syllable	The student read a word without saying one or more of its sounds, but the examiner does not mark it as incorrect
30	Saying the silent consonant	The student pronounces the silent letter in a word, but the examiner does not mark it as incorrect
31	Alterations of numbers	The student reads a different number than the one in the passage, but the examiner does not mark it as incorrect. The student also might say the correct number written in the passage, but the examiner mistakenly marks it as incorrect
32	Changed emphasis	The student places emphasis on the wrong part of a word, thus reading it incorrectly, but the examiner does not mark it as incorrect
33	Insert word	The student inserts a word, which is a time penalty only, but the examiner marks a word before or after the insertion as incorrect
34	Skipped word	The student skips a word while reading, but the examiner does not mark it as incorrect
35	Changing order	The student reverses the order of the words, but the examiner does not mark this as an insertion before (time penalty only) and skip after (incorrect)
36	Unclassifiable	The examiner marks a word as read incorrectly, but there is no apparent reason why the examiner thought a mistake was made
37	Around miscue	The student reads a word incorrectly, and the examiner marks one or more words around the original miscue as incorrect. This can be in place of the actual miscue or in addition to the actual miscue