## Abstract

In practice, risk forecasts are obtained by risk measures based on a given probability measure on a measurable space. In our study, we consider the probability measures as alternative scenarios, which refer to, for instance, different distribution assumptions, models, or economic situations. Using an improper probability measure can affect risk forecasting and lead to wrong financial decisions. In this context, we propose a Deviation-based approach for quantifying model risk associated with choosing an inappropriate probability measure for risk forecasting. This measuring approach provides us with information about how far our risk measurement process could be affected by model risk. We provide examples of Deviation-based model risk measures defined in the literature. Moreover, we are proposing new alternatives to quantify model risk, for example, Gini and Extended Gini-type model risk measures. We provide a practical example using Value-at-risk (VaR) and Expected Shortfall forecasting to illustrate our approach. Our results indicate that using an inadequate probability measure (distribution assumptions) can largely affect risk forecasting. We verify that model risk estimates present skewness and heavy tail, have significant auto-correlation and do increase in periods that coincide with the highest variability of returns.

This is a preview of subscription content, access via your institution.

## Notes

- 1.
The literature focuses mainly on the comparison of candidate probability measures when S refers to candidate models. We refer to Kuester et al. (2006), Righi and Ceretta (2015), Müller and Righi (2018) and Bianchi and Sorrentino (2020) and the references therein as examples of studies that perform the comparison of alternative strategies to forecast risk measures. For elicitable risk measures the performance comparisons from alternative models is possible (see Gneiting (2011) and Ziegel (2016)). However, classic measures in the literature, such as Expected Shortfall, are not elicitable. Besides that, models are genarally attempts for simplification of reality and it must be recognized that there is no perfect model (see Daníelsson (2008) and Reserve (2011)).

- 2.
It is also possible to apply risk measures directly in a loss or error function, from some estimation or forecasting procedure. Details are in Bignozzi and Tsanakas (2016) and Detering and Packham (2016). Müller and Righi (2019) name this approach as loss function. The measures from this approach assess the precision of individual models to forecast a particular functional. For this reason, loss function measures can be applied as a complementary criterion for forecast model selection.

- 3.
Regulatory arbitrage, in the sense used, refers to two institutions with the same portfolio that use different internal models, approved by the regulator, and therefore quantify different amounts of capital requirement. As they keep the same portfolio, they are required to hold the same or at least almost the same amount of regulatory capital. See Kellner and Rösch (2016).

- 4.
\(\varOmega \) is a no-empty set for the possible states of the world, while \(\mathcal {F}\) is a family form subsets of \(\varOmega \) closed under operations of countable unions and complements, by representing the possible events.

- 5.
In probability and statistics, a random variable is a random quantity whose values depend on outcomes of a random phenomenon. In the context of the present paper, the value of a random variable depends on the outcome of the event \(\omega \) within the underlying space \(\varOmega \).

- 6.
The expected value is the arithmetic mean of a large number of independent realizations of a random variable.

- 7.
The cumulative distribution function gives the probability that a random variable is less than or equal to a given value.

- 8.
Practically, the relative entropy is a statistical measure that describes how one given probability distribution differs from a second.

- 9.
The covariance is a well-known measure, in probability theory and statistics, which quantifies the joint variability of two random variables.

- 10.
We say a function

*f*is measurable with respect to a sigma-algebra \(\mathcal {S}\) if its information is contained on such a set, i.e., \(f^{-1}(B)\in \mathcal {S}\) for any Borel set*B*. - 11.
We use these distribution-based risk measures because they are the most commonly used in the financial industry.

- 12.
We interchangeably use the term returns and log-returns.

- 13.
In the risk forecasting literature, AR(

*p*)-GARCH(*q*,*l*) is a frequently employed model for forecasting risk measures, and, besides this, there is evidence that points to the superior performance of the GARCH model against other models. See Zhou and Anderson (2012), Degiannakis et al. (2013), and Diaz et al. (2017). We consider different specifications of the probability function*F*for \(z_t\) because, in a practical sense, risk measures are estimated from data distributions. We want to check that the risk forecasts are sensitive to distributional assumptions and whether these different assumptions generate model risk. Furthermore, our intention here is to illustrate deviation-based model risk measures and not comparing different models for predicting risk measures. However, we emphasize that other models could be considered for forecasting risk measures, such as quantile regression, stochastic volatility, and multifractal models, for instance [for references, see Koenker and Bassett (1978), Calvet and Fisher (2004), Segnon et al. (2017), and Lux and Segnon (2012). We are grateful to an anonymous reviewer for the latter mentioned remark and references.] Although our empirical analysis is univariate, extensions to the multivariate case are operable. We refer to studies such as Weiß (2013) and Müller and Righi (2018) for a comparison of alternative models in a multivariate context, regarding financial risk forecasting. We chose the univariate estimation for simplicity and since it is typically used in illustrative examples in the framework of model risk measurement. See Danielsson et al. (2016) and Kellner and Rösch (2016). - 14.
This rolling window estimation is regular in risk forecasting studies. Further, some studies show this number of observations is a good sample size for daily data. See Righi and Ceretta (2016).

- 15.
Consider a financial return distribution that have a fully parametric location-scale specification based on the expectation, dispersion and random component, represented by \(X_t = \mu _t + \sigma _tz_t\), where \(\mu _t\) and \(\sigma _t\) are mensurable considering the information available up to \( t-1 \), and \(z_t\) are i.i.d. innovations, which can assume different probability distribution functions. Assume that \(\rho \) refers to risk measure, such as VaR and ES, being \(\rho ^*(X) = -\rho (X)\). Under this specification, risk measures become \(\rho ^*(X_t) = \rho ^*(\mu _t + \sigma _tz_t) = -\mu _t + \sigma _t\rho ^*(z_t)\). Thus, using a conditional model, we first filter \(\mu _t\) and \(\sigma _t\), and, then we get \(\rho ^*\), for a given distribution. For more details, see McNeil et al. (2015).

- 16.
We do not use Extended Gini-type model risk measure (expression (23)) in our illustration because it is a general form of Gini-type model risk measure for \(r=2\). We suggest that future work should be conducted in this sense to investigate how to determine the degree of model risk-aversion according to each profile. See Berkhouch et al. (2018) for an illustration of the use of Extended Gini in the framework of market risk measurement.

- 17.
Ljung-Box test considers as null hypothesis (\(\mathcal {H}_0\)) that auto-correlations up to lag

*k*are zero, and as alternative hypothesis (\(\mathcal {H}_1\)) that auto-correlations up to lag*k*differ from zero.

## References

Acerbi, C. (2002). Spectral measures of risk: A coherent representation of subjective risk aversion.

*Journal of Banking and Finance*,*26*(7), 1505–1518.Adam, A., Houkari, M., & Laurent, J.-P. (2008). Spectral risk measures and portfolio selection.

*Journal of Banking and Finance*,*32*(9), 1870–1882.Artzner, P., Delbaen, F., Eber, J.-M., & Heath, D. (1999). Coherent measures of risk.

*Mathematical Finance*,*9*(3), 203–228.Bannör, K. F., & Scherer, M. (2013). Capturing parameter risk with convex risk measures.

*European Actuarial Journal*,*3*(1), 97–132.Barrieu, P., & Scandolo, G. (2015). Assessing financial model risk.

*European Journal of Operational Research*,*242*(2), 546–556.Basel Committee on Banking Supervision (2013). Fundamental review of the trading book: A revised market risk framework. Consultative Document, October.

Berkhouch, M., Lakhnati, G., & Righi, M. B. (2018). Extended gini-type measures of risk and variability.

*Applied Mathematical Finance*,*25*(3), 295–314.Bianchi, M. L., & Sorrentino, A. M. (2020). Measuring covar: An empirical comparison.

*Computational Economics*,*55*(2), 511–528.Bignozzi, V., & Tsanakas, A. (2016). Parameter uncertainty and residual estimation risk.

*Journal of Risk and Insurance*,*83*(4), 949–978.Breuer, T., & Csiszár, I. (2016). Measuring distribution model risk.

*Mathematical Finance*,*26*(2), 395–411.Calvet, L. E., & Fisher, A. J. (2004). How to forecast long-run volatility: Regime switching and the estimation of multifractal processes.

*Journal of Financial Econometrics*,*2*(1), 49–83.Ceriani, L., & Verme, P. (2012). The origins of the Gini index: Extracts from variabilità e mutabilità (1912) by Corrado Gini.

*The Journal of Economic Inequality*,*10*(3), 421–443.Cont, R. (2006). Model uncertainty and its impact on the pricing of derivative instruments.

*Mathematical Finance*,*16*(3), 519–547.Daníelsson, J. (2008). Blame the models.

*Journal of Financial Stability*,*4*(4), 321–328.Danielsson, J., James, K. R., Valenzuela, M., & Zer, I. (2016). Model risk of risk models.

*Journal of Financial Stability*,*23*, 79–91.Danielsson, J., & Shin, H. S. (2003).

*Endogenous risk*. Modern risk management: A history (pp. 297–316).Degiannakis, S., Floros, C., & Dent, P. (2013). Forecasting value-at-risk and expected shortfall using fractionally integrated models of conditional volatility: International evidence.

*International Review of Financial Analysis*,*27*, 21–33.Delbaen, F. (2012). Monetary Utility Functions. Osaka University CSFI lecture notes series.

Detering, N., & Packham, N. (2016). Model risk of contingent claims.

*Quantitative Finance*,*16*(9), 1357–1374.Diaz, A., Garcia-Donato, G., & Mora-Valencia, A. (2017). Risk quantification in turmoil markets.

*Risk Management*,*19*(3), 202–224.Dotson, D., McTaggart, R., Daroczi, G., & Leung, C. (2019). Package Quandl. Comprehensive R Archive Network.

Dowd, K. (2007).

*Measuring market risk*. Hoboken: Wiley.Emmer, S., Kratz, M., & Tasche, D. (2015). What is the best risk measure in practice? A comparison of standard measures.

*Journal of Risk,**18*(2).Reserve, Federal. (2011).

*Supervisory guidance on model risk management*. Office of the Comptroller of the Currency: Board of governors of the federal reserve system.Föllmer, H., & Schied, A. (2002). Convex measures of risk and trading constraints.

*Finance and Stochastics*,*6*(4), 429–447.Föllmer, H., & Schied, A. (2016).

*Stochastic Finance: An Introduction in discrete time*. De Gruyter Textbook: De Gruyter.Furman, E., Wang, R., & Zitikis, R. (2017). Gini-type measures of risk and variability: Gini shortfall, capital allocations, and heavy-tailed risks.

*Journal of Banking and Finance*,*83*, 70–84.Gini, C. (1921). Measurement of inequality of incomes.

*The Economic Journal*,*31*(121), 124–126.Giorgi, G. M. (1993). A fresh look at the topical interest.

*Metron: International Journal of Statistics*,*51*(1–2), 83–98.Giorgi, G. M. (2005). Bibliographic portrait of the Gini concentration ratio. Technical report, University Library of Munich, Germany.

Glasserman, P., & Xu, X. (2014). Robust risk measurement and model risk.

*Quantitative Finance*,*14*(1), 29–58.Gneiting, T. (2011). Making and evaluating point forecasts.

*Journal of the American Statistical Association*,*106*(494), 746–762.Henryk, G., & Silvia, M. (2006). On a relationship between distorted and spectral risk measures. Working Paper.

Jokhadze, V., Schmidt, W. M., et al. (2020). Measuring model risk in financial risk management and pricing.

*International Journal of Theoretical and Applied Finance (IJTAF)*,*23*(02), 1–37.Kellner, R., & Rösch, D. (2016). Quantifying market risk with value-at-risk or expected shortfall? Consequences for capital requirements and model risk.

*Journal of Economic Dynamics and Control*,*68*, 45–63.Kerkhof, J., Melenberg, B., & Schumacher, H. (2010). Model risk and capital reserves.

*Journal of Banking and Finance*,*34*(1), 267–279.Koenker, R., & Bassett Jr, G. (1978). Regression quantiles. Econometrica: Journal of the Econometric Society. 46: 33–50.

Krajcovicova, Z., Perez-Velasco, P. P., & Vázquez Cendón, C. (2019). A new approach to the quantification of model risk for practitioners.

*Journal of Computational Finance,**23*(2).Kuester, K., Mittnik, S., & Paolella, M. S. (2006). Value-at-risk prediction: A comparison of alternative strategies.

*Journal of Financial Econometrics*,*4*(1), 53–89.Lux, T., & Segnon, M. (2018). Multifractal models in finance: Their origin, properties and applications. The Oxford Handbook of Computational Economics and Finance.

Mao, T., & Wang, R. (2018). Risk aversion in regulatory capital principles. Working Paper.

Markowitz, H. (1952). Portfolio selection.

*The Journal of Finance*,*7*(1), 77–91.McNeil, A. J., Frey, R., & Embrechts, P. (2015).

*Quantitative risk management: concepts, techniques and tools-revised edition*. New Jersey: Princeton University Press.Müller, F. M., & Righi, M. B. (2018). Numerical comparison of multivariate models to forecasting risk measures.

*Risk Management*,*20*(1), 29–50.Müller, F. M., & Righi, M. B. (2019). Model risk measures: A review and new proposals on risk forecasting. Working Paper.

Pflug, G., & Römisch, W. (2007). Modeling. Measuring and managing risk: World Scientific.

Righi, M., & Ceretta, P. (2016a). Shortfall deviation risk: An alternative to risk measurement.

*Journal of Risk*,*19*(2), 81–116.Righi, M. B. (2018). A composition between risk and deviation measures.

*Annals of Operations Research,*1–15.Righi, M. B. (2018). A theory for combinations of risk measure. Available at: arXiv:1807.01977.

Righi, M. B., & Ceretta, P. S. (2015). A comparison of expected shortfall estimation models.

*Journal of Economics and Business*,*78*, 14–47.Righi, M. B., & Ceretta, P. S. (2016b). On the existence of an optimal estimation window for risk measures.

*Economics Bulletin*,*36*(1), 1–9.Rockafellar, R. T., Uryasev, S., & Zabarankin, M. (2006). Generalized deviations in risk analysis.

*Finance and Stochastics*,*10*(1), 51–74.Segnon, M., Lux, T., & Gupta, R. (2017). Modeling and forecasting the volatility of carbon dioxide emission allowance prices: A review and comparison of modern volatility models.

*Renewable and Sustainable Energy Reviews*,*69*, 692–704.Shalit, H., & Yitzhaki, S. (1984). Mean-gini, portfolio theory, and the pricing of risky assets.

*The Journal of Finance*,*39*(5), 1449–1468.Taylor, J. W. (2019). Forecast combinations for value at risk and expected shortfall. International Journal of Forecasting, In Press.

Wang, R., Wei, Y., & Willmot, G. (2018). Characterization, robustness and aggregation of signed Choquet integrals. Working Paper.

Weiß, G. N. F. (2013). Copula-GARCH versus dynamic conditional correlation: An empirical study on VaR and ES forecasting accuracy.

*Review of Quantitative Finance and Accounting*,*41*(2), 179–202.Yitzhaki, S. (1983). On an extension of the Gini inequality index.

*International Economic Review*,*24*(3), 617–628.Yitzhaki, S. (1998). More than a dozen alternative ways of spelling Gini. In Research in economic inequality.

Yitzhaki, S., & Schechtman, E. (2005). The properties of the extended gini measures of variability and inequality. Available at SSRN 815564.

Yitzhaki, S., & Schechtman, E. (2012).

*The gini methodology: A primer on a statistical methodology*. Berlin, Newyork: Springer. in Statistics.Zhou, J., & Anderson, R. I. (2012). Extreme risk measures for international reit markets.

*The Journal of Real Estate Finance and Economics*,*45*(1), 152–170.Ziegel, J. F. (2016). Coherence and elicitability.

*Mathematical Finance*,*26*(4), 901–918.

## Author information

### Affiliations

### Corresponding author

## Ethics declarations

### Conflict of interest

The authors report no conflict of interest.

## Additional information

### Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

## About this article

### Cite this article

Berkhouch, M., Müller, F.M., Lakhnati, G. *et al.* Deviation-Based Model Risk Measures.
*Comput Econ* (2021). https://doi.org/10.1007/s10614-021-10093-x

Accepted:

Published:

### Keywords

- Model risk
- Model risk measurement
- Deviation measures
- Deviation-based approach
- Robust finance