Advertisement

Industrial-Scale Ad Hoc Risk Analytics Using MapReduce

  • Andrew Rau-ChaplinEmail author
  • Zhimin Yao
  • Norbert Zeh
Chapter
First Online:
  • 3.4k Downloads
Part of the Studies in Big Data book series (SBD, volume 16)

Abstract

Modern reinsurance companies hold portfolios consisting of thousands of reinsurance contracts covering millions of individually insured locations. To ensure capital adequacy and for fine-grained financial planning, these companies carry out large-scale Monte Carlo simulations to estimate the probabilities that the losses incurred due to catastrophic events such as hurricanes, earthquakes, etc. exceed certain critical values. This is a computationally intensive process that requires the use of parallelism to answer risk queries over a portfolio in a timely manner. We present a system that uses the MapReduce framework to evaluate risk analysis queries on industrial-scale portfolios efficiently. In contrast to existing production systems, this system is designed to support arbitrary ad hoc queries an analyst may pose while achieving a performance that is very close to that of highly optimized production systems, which often only support evaluating a limited set of risk metrics. For example, a full portfolio risk analysis run consisting of a 1,000,000-trial simulation, with 1,000 events per trial, and 3,200 risk transfer contracts can be completed on a 16-node Hadoop cluster in just over 20 min. MapReduce is an easy-to-use parallel programming framework that offers the flexibility required to develop the type of system we describe. The key to nearly matching the performance of highly optimized production systems was to judiciously choose which parts of our system should depart from the classical MapReduce model and use a combination of advanced features offered by Apache Hadoop with carefully engineered data structure implementations to eliminate performance bottlenecks while not sacrificing the flexibility of our system.

Keywords

Loss Distribution MapReduce Framework Query Engine Distribute File System Risk Metrics 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

References

  1. 1.
    de Alba, E., Zúñiga, J., Corzo, M.A.R.: Measurement and transfer of catastrophic risk. ASTIN Bull. 40(2), 547–568 (2010)MathSciNetGoogle Scholar
  2. 2.
    Amazon Elastic MapReduce (Amazon EMR). http://aws.amazon.com/elasticmapreduce. Accessed 25 May 2013
  3. 3.
    Anderson, R.R., Dong, W.: Pricing catastrophe reinsurance with reinstatement provisions using a catastrophe model. In: Casualty Actuarial Society Forum, pp. 303–322 (Summer 1998)Google Scholar
  4. 4.
    Apache Hadoop. http://hadoop.apache.org. Accessed 25 May 2013
  5. 5.
    Bahl, A.K., Baltzer, O., Rau-Chaplin, A., Varghese, B.: Parallel simulations for analysing portfolios of catastrophic event risk. In: Proceedings of the International Supercomputing Conference (SC12). Workshop on High Performance Computational Finance, pp. 1176–1184. Salt Lake City, Utah, USA (Oct 2012)Google Scholar
  6. 6.
    Berens, R.M.: Reinsurance contracts with a multi-year aggregate limit. In: Casualty Actuarial Society Forum, pp. 289–308 (Spring 1997)Google Scholar
  7. 7.
    Byrne, M., Dehne, F., Hickey, G., Rau-Chaplin, A.: Parallel catastrophe modelling on a Cell B.E. J. Parallel Emergent Distrib. Syst. 25(5), 401–410 (2010)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Capriolo, E., Wampler, D., Rutherglen, J.: Programming Hive, 1st edn. O’Reilly Media (2012)Google Scholar
  9. 9.
    Castella, H., de Montmollin, G., Rüttener, E.: Catastrophe Portfolio Modeling: A Complete View. PartnerRe (2009)Google Scholar
  10. 10.
    Cloudera. http://www.cloudera.com. Accessed 14 June 2015
  11. 11.
    Coelho, M., Rau-Chaplin, A.: eXsight: An Analytical Framework for Quantifying Financial Loss in the Aftermath of Catastrophic Events. In: Proceedings of the Workshop ISSASiM (DEXA 2014), Munich, Germany (2014)Google Scholar
  12. 12.
    Condie, T., Conway, N., Alvaro, P., Hellerstein, J.M., Elmeleegy, K., Sears, R.: MapReduce online. EECS Department, University of California, Berkeley, Technical Report No. UCB/EECS-2009-136, October 2009Google Scholar
  13. 13.
    Cortes, O.A.C., Rau-Chaplin, A., Wilson, D., Cook, I., Gaiser-Porter, J.: Efficient optimization of reinsurance contracts using discretized PBIL. In: International Conference on Data Analytics (Data Analytics 2013), Porto, Portugal, pp. 18–24 (2013)Google Scholar
  14. 14.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)Google Scholar
  15. 15.
    Dong, W., Shah, H., Wong, F.: A rational approach to pricing of catastrophe insurance. J. Risk Uncertainty 12, 201–218 (1996)CrossRefzbMATHGoogle Scholar
  16. 16.
    Eaton, C., Deroos, D., Deutsch, T., Lapis, G., Zikopoulos, P.: Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data. McGraw Hill (2012)Google Scholar
  17. 17.
    Eden, R.: GNU Trove: High Performance Collections for Java. http://trove.starlight-systems.com. Accessed 19 Jan 2013
  18. 18.
    Gaivoronski, A.A., Pflug, G.: Value-at-risk in portfolio optimization: properties and computational approach. J. Risk 9(2), 1–31 (Winter 2004–2005)Google Scholar
  19. 19.
    Glasserman, P., Heidelberger, P., Shahabuddin, P.: Portfolio value-at-risk with heavy-tailed risk factors. Math. Finance 12(3), 239–269 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
  21. 21.
    Grossi, P., Kunreuter, H.: Catastrophe Modelling: A New Approach to Managing Risk. Springer (2005)Google Scholar
  22. 22.
    Hadoop Distributed File System. http://hadoop.apache.org/docs/r1.0.4/hdfs_design.htmlS. Accessed 25 May 2013
  23. 23.
    Harrison, C.: Reinsurance Principles and Practices. American Institute for Charter Property Casualty Underwriters (2008)Google Scholar
  24. 24.
    HiveQL. http://hive.apache.org. Accessed 25 May 2013
  25. 25.
    Lee, K.-H., Lee, Y.-J., Choi, H., Chung, Y.D., Moon, B.: Parallel data processing with MapReduce: a survey. SIGMOD Rec. 40(4), 11–20 (2011)CrossRefGoogle Scholar
  26. 26.
    Meyers, G.G., Klinker, F.L., Lalonde, D.A.: The aggregation and correlation of reinsurance exposure. In: Casualty Actuarial Society Forum, pp. 69–152 (Spring 2003)Google Scholar
  27. 27.
    Oracle. Java Platform SE 7 HashMap. http://docs.oracle.com/javase/7/docs/api/java/util/HashMap.html. Accessed 28 Jan 2014
  28. 28.
    Osiaski, S., Weiss, D.: HPPC: High Performance Primitive Collections for Java. http://labs.carrotsearch.com/hppc.html. Accessed 19 Jan 2013
  29. 29.
    Rocks Cluster Distribution. http://www.rocksclusters.org/. Accessed 14 June 2015
  30. 30.
    Rau-Chaplin, A., Varghese, B.: Accounting for secondary uncertainty: efficient computation of portfolio risk measures on multi and many core architectures. In: Proceedings of the 6th Workshop on High Performance Computational Finance (WHPCF), Denver, USA, No. 3, pp. 1–10 (2013)Google Scholar
  31. 31.
    Rau-Chaplin, A., Varghese, B., Yao, Z.: A MapReduce framework for analysing portfolios of catastrophic risk with secondary uncertainty. In: Proceedings of the Workshop of the International Conference on Computational Science (2013)Google Scholar
  32. 32.
    Rau-Chaplin, A., Varghese, B., Wilson, D., Yao, Z., Zeh, N.: QuPARA: query-driven large-scale portfolio aggregate risk analysis on MapReduce. In: Proceedings of the IEEE International Conference on Big Data (IEEE BigData 2013), IEEE Comp. Soc. Dig. Library (2013)Google Scholar
  33. 33.
    Shvachko, K., Hairong, K., Radia, S., Chansler, R.: The Hadoop distributed file system. In: Proceedings of the 26th IEEE Symposium on Mass Storage Systems and Technologies, pp. 1–10 (2010)Google Scholar
  34. 34.
    Salcedo-Sanz, S., Carro-Calvo, L., Claramunt, M., Castañer, A., Mármol, Maite: Effectively tackling reinsurance problems by using evolutionary and swarm intelligence algorithms. Risks 2(2), 132–145 (2014)CrossRefGoogle Scholar
  35. 35.
    White, T.: Hadoop: The Definitive Guide, 1st edn. O’Reilly Media (2009)Google Scholar
  36. 36.
    Wilkinson, M.E.: Estimating probable maximum loss with order statistics, pp. 195–209. Casualty Actuarial Society, Forum (1982)Google Scholar
  37. 37.
    Woo, G.: Natural catastrophe probable maximum loss. Br. Actuarial J. 8(5), 943–959 (2002)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Risk Analytics LabDalhousie UniversityHalifaxCanada

Personalised recommendations

Cite chapter

Buy options