Dear Editors,

We welcome Dr. Naderi’s comments [2] regarding our use of kappa to calculate inter-observer reliability. As Dr. Naderi correctly points out, the proportion of concordant and discordant cells can affect the kappa value. In our study, kappa values were calculated using 2 × 2 tables, which takes this into account. In response to the letter, we recalculated weighted kappas and found that the results were exactly the same as those quoted in the original paper. SAS (Statistics Analysis System) documentation also supports this in the following statement: “The weighted kappa coefficient is a generalisation of the simple kappa coefficient that uses weights to quantify the relative difference between categories. For 2 x 2 tables, the weighted kappa coefficient equals the simple kappa coefficient” [2].

Despite this, we do recognise the limitations in using kappa to assess reliability and therefore also included percentage agreement and discussed these limitations in the paper. While these limitations are valid, the results still highlight that inter-reporter agreement is suboptimal and efforts should be made to improve the reliability of reporting.