Measures of a Rating’s Discriminative Power: Applications and Limitations

Abstract

A key attribute of a rating system is its discriminative power, i.e., its ability to separate good credit quality from bad credit quality. Similar problems arise in other scientific disciplines. In medicine, the quality of a diagnostic test is mainly determined by its ability to distinguish between ill and healthy persons. Analogous applications exist in biology, information technology, and engineering sciences. The development of measures of discriminative power dates back to the early 1950s. An interesting overview is given in Swets (1988).

Appendices

Appendix A. Proof of (13.2)

We introduce the shortcut notation $\hat{C}_D^i = {\hat{C}_D}\left( {{R_i}} \right)$, $\hat{C}_{ND}^i$and $\hat{C}_T^i$have a similar meaning. Furthermore, we denote the sample default probability by $\hat{p}$. Note that $\hat{C}_T^i$can be written in terms of $\hat{C}_{ND}^i$and $\hat{C}_D^i$as

$$ \hat{C}_T^i = \hat{p} \cdot \hat{C}_D^i + \left( {1 - \hat{p}} \right) \cdot \hat{C}_{ND}^i. $$

(13.23)

By computing simple integrals, we find for AUROC, a _R  + 0.5, and a _P the expressions

$$ \begin{array}{llllll} AUROC = \sum\limits_{i = 1}^k {0.5 \cdot \left( {\hat{C}_D^i + \hat{C}_D^{i - 1}} \right) \cdot \left( {\hat{C}_{ND}^i - \hat{C}_{ND}^{i - 1}} \right)}, \\{a_R} + 0.5 = \sum\limits_{i = 1}^k {0.5 \cdot \left( {\hat{C}_D^i + \hat{C}_D^{i - 1}} \right) \cdot \left( {\hat{C}_T^i - \hat{C}_T^{i - 1}} \right)}, \\{a_P} = 0.5 \cdot \left( {1 - \hat{p}} \right). \\\end{array} $$

(13.24)

Plugging (13.23) into the expression for ${a_R} + 0.5$and simplifying leads to

$$ \begin{array}{llllll} {a_R} + 0.5 = \sum\limits_{i = 1}^k {0.5 \cdot \left( {\hat{C}_D^i + \hat{C}_D^{i - 1}} \right) \cdot \left( {\hat{C}_T^i - \hat{C}_T^{i - 1}} \right)} \\ = \sum\limits_{i = 1}^k {0.5 \cdot \left( {\hat{C}_D^i + \hat{C}_D^{i - 1}} \right) \cdot \left( {\hat{p} \cdot \left( {\hat{C}_D^i - \hat{C}_D^{i - 1}} \right) + \left( {1 - \hat{p}} \right) \cdot \left( {\hat{C}_{ND}^i - \hat{C}_{ND}^{i - 1}} \right)} \right)} \\ = \left( {1 - \hat{p}} \right) \cdot \sum\limits_{i = 1}^k {0.5 \cdot \left( {\hat{C}_D^i + \hat{C}_D^{i - 1}} \right) \cdot \left( {\hat{C}_{ND}^i - \hat{C}_{ND}^{i - 1}} \right)} \\ \quad+ \hat{p} \cdot \sum\limits_{i = 1}^k {0.5 \cdot \left( {\hat{C}_D^i + \hat{C}_D^{i - 1}} \right) \cdot \left( {\hat{C}_D^i - \hat{C}_D^{i - 1}} \right)} \\ = \left( {1 - \hat{p}} \right) \cdot AUROC + 0.5 \cdot \hat{p} \cdot \sum\limits_{i = 1}^k {\left( {{{\left( {\hat{C}_D^i} \right)}^2} - {{\left( {\hat{C}_D^{i - 1}} \right)}^2}} \right)} \\ = \left( {1 - \hat{p}} \right) \cdot AUROC + 0.5 \cdot \hat{p}{ }. \\\end{array} $$

(13.25)

Taking (13.24) and (13.25) together leads to the desired result

$$ AR = \frac{{{a_R}}}{{{a_P}}} = \frac{{\left( {1 - \hat{p}} \right) \cdot \left( {AUROC - 0.5} \right)}}{{0.5 \cdot \left( {1 - \hat{p}} \right)}} = 2 \cdot AUROC - 1. $$

Appendix B. Proof of (13.7)

Using the same shortcut notation as in Appendix A, we get

$$ \begin{array}{llllll} AUROC = \sum\limits_{i = 1}^k {0.5 \cdot \left( {\hat{C}_D^i + \hat{C}_D^{i - 1}} \right) \cdot \left( {\hat{C}_{ND}^i - \hat{C}_{ND}^{i - 1}} \right)} \\ = \sum\limits_{i = 1}^k {0.5 \cdot \left( {P\left( {{{\hat{S}}_D} \le {R_i}} \right) + P\left( {{{\hat{S}}_D} \le {R_{i - 1}}} \right)} \right) \cdot P\left( {{{\hat{S}}_{ND}} = {R_i}} \right)} \\ = \sum\limits_{i = 1}^k {\left( {P\left( {{{\hat{S}}_D} \le {R_{i - 1}}} \right) + 0.5 \cdot P\left( {{{\hat{S}}_D} = {R_i}} \right)} \right) \cdot P\left( {{{\hat{S}}_{ND}} = {R_i}} \right)} \\ = \sum\limits_{i = 1}^k {P\left( {{{\hat{S}}_D} \le {R_{i - 1}}} \right) \cdot P\left( {{{\hat{S}}_{ND}} = {R_i}} \right)} + 0.5 \cdot \sum\limits_{i = 1}^k {P\left( {{{\hat{S}}_D} = {R_i}} \right) \cdot P\left( {{{\hat{S}}_{ND}} = {R_i}} \right)} \\ = P\left( {{{\hat{S}}_D}\, < \,{{\hat{S}}_{ND}}} \right) + 0.5 \cdot P\left( {{{\hat{S}}_D} = {{\hat{S}}_{ND}}} \right) \\\end{array} $$

which proves (13.7).