Assessment Literacy of Singapore Chinese Language Teachers in Primary and Secondary Schools

Abstract

Language assessment literacy refers to language teachers’ knowledge of measurement principles, practices, and applications to classroom assessment, especially to language assessment issues. The study reports on Singapore primary and secondary Chinese Language teachers’ knowledge of language assessment. The survey involved 323 primary and secondary teachers who responded to the Assessment Literacy Scale prepared by the researchers. Results at the subtest and item levels revealed the teachers’ shortfalls in assessment literacy and indicated their training needs. The respondents were able to correctly answer general questions of commonsensical nature but were weak where specific technical knowledge was concerned. Findings from this study have implications for primary and secondary Chinese Language teachers’ training in assessment literacy.

Appendix: Methodological Notes on Reliability and Validity

A report of this survey is incomplete without a discussion on two key concepts of measurement, reliability, and validity.

The conventional method of assessing score reliability is the Cronbach’s alpha coefficient which indicates the degree of internal consistency among the items, with the assumption that the items are homogeneous and the sample is heterogeneous. The 40 items of the Assessment Literacy Scale crafted for this study are scored 1 (right) or 0 (wrong).

Richardson Formula 20 (KR20), which is an equivalent of Cronbach’s alpha but for dichotomous items, was used.

Table 6.6 below shows the KR20 reliabilities for the four subscales and the scale as a whole. It is clear that the KR20 reliabilities are disappointingly low by the conventional expectation and this may lead to the question of trustworthiness of the survey results.

Table 6.6 KR20 reliabilities

However, there have been criticisms, among others, on Cronbach’s alpha as a measure of item homogeneity or unidimensionality. For example, Sijtsma (2009: para. 4.2) commented:

There is no clear and unambiguous relationship between alpha and the internal structure of a test. This can be demonstrated in a simple way. First, it is shown that a 1-factor test may have any alpha value. Thus, it may be concluded that the value of alpha says very little if anything about unidimensionality. Second, it is shown that different tests of varying factorial composition may have the same alpha value. Thus, it may be concluded that alpha says very little if anything about multiple-factor item structures.

One factor leading to the low reliabilities as shown in Table 6.6 is the heterogeneous nature of item content among the 40 items of the Assessment Literacy Scale as they cover many different aspects of educational measurement, some qualitative and other quantitative in nature, even within a particular subtest. This being the case, it renders the conventional reliability measures (i.e., Cronbach’s alpha and its equivalent KR20) which assume item homogeneity unsuitable for the purpose of the present study. Thus, the KR20 reliabilities routinely calculated and presented in Table 6.6 are not to be taken seriously.

Another factor contributing to low-score reliability is group homogeneity. Pike and Hudson (No date: para. 1) discussed the limitation of using Cronbach’s alpha to estimate reliability when using a sample with homogeneous responses in the measured construct and described the risk of falsely concluding that a new instrument may have poor reliability and demonstrates the use of an alternate statistic that may serve as a cushion against such errors. The authors recommended the calculation of the relative alpha by considering the ratio between the standard error of measurement (SEM) which itself involves the reliability as shown in the formula; thus,

$$ \mathrm{S}\mathrm{E}\mathrm{M}=\mathrm{S}\mathrm{D}*\mathrm{SQRT}\left(1 - \mathrm{reliability}\right) $$

The relative alpha which can take a value between 0.0 and 1.0 indicates the extent to which the scores can be trusted, in a sense, an alternative way to evaluate score reliability. The formula is

$$ \mathrm{Relative}\ \mathrm{Alpha}=1\hbox{--}\ {\mathrm{SEM}}^2/\ {\left(\mathrm{Range}/6\right)}^2 $$

In this formula, SEM is the usual indicator of the lack of trustworthiness of the obtained scores and, under normal circumstances, the scores for a scale will theoretically span over six standard deviations. Thus, the second term on the right is an indication of the proportion of the test variance that is unreliable. With these, relative alpha indicates the proportion of test variance offset for its unreliable portion, i.e., the proportion of test variance which can be trusted.

In the present study, the maximum possible score for the Assessment Literacy Scale is 40 and the theoretically possible standard deviation is 6.67 = 40/6. However, the actual data yields for the Scale as a whole standard deviations of 4.24 (primary) and 4.66 (secondary), which are 0.64 and 0.70, respectively, of the theoretical standard deviations. In other words, the two groups are found to be more homogeneous than expected.

Table 6.7 shows the relative alphas for the primary and secondary groups of Chinese Language teachers surveyed here. The statistics suggest that much of the test variance has been captured by the 40-item Assessment Literacy Scale and the scores can therefore be trusted.

Table 6.7 Relative alpha coefficients

As for validity, it required information beyond the test scores. Ideally, the criterion scores for validity can come from a test of application of measurement concepts and techniques, but such information is not available within the survey results, although some of the 40 items of the Assessment Literacy Scale are of this type, for instance, those items on statistical concepts. However, indirect evidence of the score validity is provided by the teachers’ responses to the open-ended question asking for comments and suggestions with regard to educational assessment.

For the open-ended question, the primary Chinese teachers made 14 responses and the secondary teachers 22, totally 36 responses. Most of the responses (primary 7 and secondary 12) reflect the teachers’ realization that assessment plays an important role in their teaching for which specialized knowledge is needed. Examples of such responses are shown below:

• What is taught and what is assessed should be consistent.

• Teachers need to have knowledge of educational measurement.

• Need to popularize knowledge of assessment among the school leaders.

• Hope to gain knowledge of educational measurement so that I can assess with in-depth understanding.

• Without knowledge of educational measurement, data analysis of results conducted in the school is superficial.

• Very much needed!

• Will help improving teaching.

A second type of responses reflect the difficulty the teachers had in understanding items which involve technical concepts and terminologies (primary 4, secondary 7). Such responses are expected in view of the lack of more formal and intensive training in educational assessment. Examples of such responses are shown below:

• Not familiar with the technical terms.

• Too many technical, I don’t understand.

• I don’t understand some of the questions.

• Many mathematical terms; I don’t understand.

The third type of responses reflect the need of the teachers to be convinced that assessment training is necessary for them to use assessment results properly as part of instruction (primary 3, secondary 3). Examples of such responses are shown below:

• Can assessment really raise the students’ achievement and attitude? Will it add on the teachers’ work? Really helpful to the students?

• Does data help in formative assessment?

The responses reaffirm the test-taking attitude of the teachers when responding to the Assessment Literacy Scale. The seriousness with which they completed the survey is clearly evident. The second type of responses corroborates with the finding that they lack relevant specific training in educational assessment and hence found the technical terms and concepts unfamiliar; this truly reflects their position and lack of knowledge. The third type of responses indicates the reservation and inquisitiveness of some of the respondents; this indirectly reflects that they need to be convinced that they need more training in educational measurement.