An Empirical Assessment of Guttman’s Lambda 4 Reliability Coefficient

Abstract

Numerous alternative indices for test reliability have been proposed as being superior to Cronbach’s alpha. One such alternative is Guttman’s L4. This is calculated by dividing the items in a test into two halves such that the covariance between scores on the two halves is as high as possible. However, although simple to understand and intuitively appealing, the method can potentially be severely positively biased if the sample size is small or the number of items in the test is large.

To begin with this paper compares a number of available algorithms for calculating L4. We then empirically evaluate the bias of L4 for 51 separate upper secondary school examinations taken in the UK in June 2012. For each of these tests we have evaluated the likely bias of L4 for a range of different sample sizes. The results show that the positive bias of L4 is likely to be small if the estimated reliability is larger than 0.85, if there are less than 25 items and if a sample size of more than 3,000 is available. A sample size of 1,000 may be sufficient if the estimate of L4 is above 0.9.

Appendix: R Code to Find Best Split Using the “Start-Then-Improve” Algorithm

• #Function to find best split half from a given starting split

• MaxSplitHalf = function(data,xal){

• #data – matrix of items scores (row=candidates,column=items)

• #xal – vector of 0s and 1s specifying initial split

• nite = ncol(data)

• cov1 = cov(data)

• v = diag(cov1)

• yal = 1-xal

• ones = rep(1,nite)

• covxy = t(xal)%*%cov1%*%yal

• #Code to examine all possible swaps

• maxchg1=9;

• while(maxchg1>0){

• #Calculate change for swapping items in X and Y;

• #This is equal to 2covxiyj+covxix+covyyj-vx-vy-covxiy-covxyj;

• covxiyj = cov1

• covxix = (cov1%*%xal)%*%t(ones)

• covyyj = ones%*%(yal%*%cov1)

• vx = v%*%t(ones)

• vy = t(vx)

• covxiy = (cov1%*%yal)%*%t(ones)

• covxyj = ones%*%(xal%*%cov1)

• result = 2*covxiyj+covxix+covyyj-vx-vy-covxiy-covxyj

• for (i in 1:nite){for (j in 1:nite){if (xal[i]==xal[j])

• {result[i,j]=0}}}

• #Add bits for swapping with no other item

• result = cbind(result,as.vector(cov1%*%xal-cov1%*%yal-v)*xal)

• result = rbind(result,c(as.vector(cov1%*%yal-cov1%*%xal-v)*yal,0))

• #find indices of maximum change;

• maxchg=0

• maxchgx=0

• maxchgy=0

• which1=which(result==max(result),arr.ind=TRUE)[1,]

• if (result[which1[1],which1[2]]>0){maxchgx=which1[1]

• maxchgy=which1[2]

• maxchg=result[which1[1],which1[2]]}

• maxchg1 = maxchg

• if (maxchgx>0 & maxchgx<(nite+1)) {xal[maxchgx]=0}

• if (maxchgy>0 & maxchgy<(nite+1)) {xal[maxchgy]=1}

• if (maxchgx>0 & maxchgx<(nite+1)) {yal[maxchgx]=1}

• if (maxchgy>0 & maxchgy<(nite+1)) {yal[maxchgy]=0}

• covxy = t(xal)%*%cov1%*%yal}

• guttman = 4*covxy/sum(cov1)

• pites = sum(xal)/nite

• raju = covxy/(sum(cov1)*pites*(1-pites))

• v1 = t(xal)%*%cov1%*%xal

• v2 = t(yal)%*%cov1%*%yal

• feldt = 4*covxy/(sum(cov1)-((v1-v2)/sqrt(sum(cov1)))**2);

• res = list(guttman=as.vector(guttman),

• raju=as.vector(raju),

• feldt=as.vector(feldt),

• xal=xal)

• return(res)}

• #Maximise L4 starting from odd/even and 12 splits from 12x12 Hadamard matrix

• library(survey)

• MaxSplitHalfHad12 = function(data){

• #data – matrix of items scores (row=candidates,column=items)

• #start with odd vs even

• nite = ncol(data)

• sequence = 1:nite

• xal = (sequence%%2)

• res1 = MaxSplitHalf(data,xal)

• #now try 12 further splits based on 12*12 Hadamard matrix

• had = hadamard(11)

• for (iz in 1:12){

• nextra = max(nite-12,0)

• resrand = MaxSplitHalf(data,c(had[,iz],rep(0,nextra))[1:nite])

• if (resrand$guttman>res1$guttman){res1 = resrand}}

• return(res1)}

• #Maximise using exhaustive search

• library(Lambda4)

• MaxSplitExhaustive = function(data){

• #data – matrix of items scores (row=candidates,column=items)

• cov1 = cov(data)

• nite = dim(data)[2]

• mat1 = (bin.combs(nite)+1)/2

• res1 = list(guttman=0,xal=rep(-99,nite))

• for (jjz in 1:length(mat1[,1])){

• xal = mat1[jjz,]

• gutt1 = 4*(t(xal)%*%cov1%*%(1-xal))/sum(cov1)

• resrand = list(guttman=gutt1,xal=xal)

• if (resrand$guttman>res1$guttman){res1 = resrand}}

• return(res1)}

• #Examples of use (using data from the Lambda4 package)

• data(Rosenberg)

• MaxSplitHalf(Rosenberg,c(0,1,0,1,0,1,0,1,0,1))

• MaxSplitHalfHad12(Rosenberg)

• MaxSplitExhaustive(Rosenberg)