Abstract
Modern reinsurance companies hold portfolios consisting of thousands of reinsurance contracts covering millions of individually insured locations. To ensure capital adequacy and for fine-grained financial planning, these companies carry out large-scale Monte Carlo simulations to estimate the probabilities that the losses incurred due to catastrophic events such as hurricanes, earthquakes, etc. exceed certain critical values. This is a computationally intensive process that requires the use of parallelism to answer risk queries over a portfolio in a timely manner. We present a system that uses the MapReduce framework to evaluate risk analysis queries on industrial-scale portfolios efficiently. In contrast to existing production systems, this system is designed to support arbitrary ad hoc queries an analyst may pose while achieving a performance that is very close to that of highly optimized production systems, which often only support evaluating a limited set of risk metrics. For example, a full portfolio risk analysis run consisting of a 1,000,000-trial simulation, with 1,000 events per trial, and 3,200 risk transfer contracts can be completed on a 16-node Hadoop cluster in just over 20 min. MapReduce is an easy-to-use parallel programming framework that offers the flexibility required to develop the type of system we describe. The key to nearly matching the performance of highly optimized production systems was to judiciously choose which parts of our system should depart from the classical MapReduce model and use a combination of advanced features offered by Apache Hadoop with carefully engineered data structure implementations to eliminate performance bottlenecks while not sacrificing the flexibility of our system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
de Alba, E., Zúñiga, J., Corzo, M.A.R.: Measurement and transfer of catastrophic risk. ASTIN Bull. 40(2), 547–568 (2010)
Amazon Elastic MapReduce (Amazon EMR). http://aws.amazon.com/elasticmapreduce. Accessed 25 May 2013
Anderson, R.R., Dong, W.: Pricing catastrophe reinsurance with reinstatement provisions using a catastrophe model. In: Casualty Actuarial Society Forum, pp. 303–322 (Summer 1998)
Apache Hadoop. http://hadoop.apache.org. Accessed 25 May 2013
Bahl, A.K., Baltzer, O., Rau-Chaplin, A., Varghese, B.: Parallel simulations for analysing portfolios of catastrophic event risk. In: Proceedings of the International Supercomputing Conference (SC12). Workshop on High Performance Computational Finance, pp. 1176–1184. Salt Lake City, Utah, USA (Oct 2012)
Berens, R.M.: Reinsurance contracts with a multi-year aggregate limit. In: Casualty Actuarial Society Forum, pp. 289–308 (Spring 1997)
Byrne, M., Dehne, F., Hickey, G., Rau-Chaplin, A.: Parallel catastrophe modelling on a Cell B.E. J. Parallel Emergent Distrib. Syst. 25(5), 401–410 (2010)
Capriolo, E., Wampler, D., Rutherglen, J.: Programming Hive, 1st edn. O’Reilly Media (2012)
Castella, H., de Montmollin, G., Rüttener, E.: Catastrophe Portfolio Modeling: A Complete View. PartnerRe (2009)
Cloudera. http://www.cloudera.com. Accessed 14 June 2015
Coelho, M., Rau-Chaplin, A.: eXsight: An Analytical Framework for Quantifying Financial Loss in the Aftermath of Catastrophic Events. In: Proceedings of the Workshop ISSASiM (DEXA 2014), Munich, Germany (2014)
Condie, T., Conway, N., Alvaro, P., Hellerstein, J.M., Elmeleegy, K., Sears, R.: MapReduce online. EECS Department, University of California, Berkeley, Technical Report No. UCB/EECS-2009-136, October 2009
Cortes, O.A.C., Rau-Chaplin, A., Wilson, D., Cook, I., Gaiser-Porter, J.: Efficient optimization of reinsurance contracts using discretized PBIL. In: International Conference on Data Analytics (Data Analytics 2013), Porto, Portugal, pp. 18–24 (2013)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)
Dong, W., Shah, H., Wong, F.: A rational approach to pricing of catastrophe insurance. J. Risk Uncertainty 12, 201–218 (1996)
Eaton, C., Deroos, D., Deutsch, T., Lapis, G., Zikopoulos, P.: Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data. McGraw Hill (2012)
Eden, R.: GNU Trove: High Performance Collections for Java. http://trove.starlight-systems.com. Accessed 19 Jan 2013
Gaivoronski, A.A., Pflug, G.: Value-at-risk in portfolio optimization: properties and computational approach. J. Risk 9(2), 1–31 (Winter 2004–2005)
Glasserman, P., Heidelberger, P., Shahabuddin, P.: Portfolio value-at-risk with heavy-tailed risk factors. Math. Finance 12(3), 239–269 (2002)
Google MapReduce. https://developers.google.com/appengine/docs/python/dataprocessing/overview. Accessed 25 May 2013
Grossi, P., Kunreuter, H.: Catastrophe Modelling: A New Approach to Managing Risk. Springer (2005)
Hadoop Distributed File System. http://hadoop.apache.org/docs/r1.0.4/hdfs_design.htmlS. Accessed 25 May 2013
Harrison, C.: Reinsurance Principles and Practices. American Institute for Charter Property Casualty Underwriters (2008)
HiveQL. http://hive.apache.org. Accessed 25 May 2013
Lee, K.-H., Lee, Y.-J., Choi, H., Chung, Y.D., Moon, B.: Parallel data processing with MapReduce: a survey. SIGMOD Rec. 40(4), 11–20 (2011)
Meyers, G.G., Klinker, F.L., Lalonde, D.A.: The aggregation and correlation of reinsurance exposure. In: Casualty Actuarial Society Forum, pp. 69–152 (Spring 2003)
Oracle. Java Platform SE 7 HashMap. http://docs.oracle.com/javase/7/docs/api/java/util/HashMap.html. Accessed 28 Jan 2014
Osiaski, S., Weiss, D.: HPPC: High Performance Primitive Collections for Java. http://labs.carrotsearch.com/hppc.html. Accessed 19 Jan 2013
Rocks Cluster Distribution. http://www.rocksclusters.org/. Accessed 14 June 2015
Rau-Chaplin, A., Varghese, B.: Accounting for secondary uncertainty: efficient computation of portfolio risk measures on multi and many core architectures. In: Proceedings of the 6th Workshop on High Performance Computational Finance (WHPCF), Denver, USA, No. 3, pp. 1–10 (2013)
Rau-Chaplin, A., Varghese, B., Yao, Z.: A MapReduce framework for analysing portfolios of catastrophic risk with secondary uncertainty. In: Proceedings of the Workshop of the International Conference on Computational Science (2013)
Rau-Chaplin, A., Varghese, B., Wilson, D., Yao, Z., Zeh, N.: QuPARA: query-driven large-scale portfolio aggregate risk analysis on MapReduce. In: Proceedings of the IEEE International Conference on Big Data (IEEE BigData 2013), IEEE Comp. Soc. Dig. Library (2013)
Shvachko, K., Hairong, K., Radia, S., Chansler, R.: The Hadoop distributed file system. In: Proceedings of the 26th IEEE Symposium on Mass Storage Systems and Technologies, pp. 1–10 (2010)
Salcedo-Sanz, S., Carro-Calvo, L., Claramunt, M., Castañer, A., Mármol, Maite: Effectively tackling reinsurance problems by using evolutionary and swarm intelligence algorithms. Risks 2(2), 132–145 (2014)
White, T.: Hadoop: The Definitive Guide, 1st edn. O’Reilly Media (2009)
Wilkinson, M.E.: Estimating probable maximum loss with order statistics, pp. 195–209. Casualty Actuarial Society, Forum (1982)
Woo, G.: Natural catastrophe probable maximum loss. Br. Actuarial J. 8(5), 943–959 (2002)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Rau-Chaplin, A., Yao, Z., Zeh, N. (2016). Industrial-Scale Ad Hoc Risk Analytics Using MapReduce. In: Japkowicz, N., Stefanowski, J. (eds) Big Data Analysis: New Algorithms for a New Society. Studies in Big Data, vol 16. Springer, Cham. https://doi.org/10.1007/978-3-319-26989-4_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-26989-4_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-26987-0
Online ISBN: 978-3-319-26989-4
eBook Packages: EngineeringEngineering (R0)