Abstract
We introduce ScannerBit, the statistics and sampling module of the public, open-source global fitting framework GAMBIT. ScannerBit provides a standardised interface to different sampling algorithms, enabling the use and comparison of multiple computational methods for inferring profile likelihoods, Bayesian posteriors, and other statistical quantities. The current version offers random, grid, raster, nested sampling, differential evolution, Markov Chain Monte Carlo (MCMC) and ensemble Monte Carlo samplers. We also announce the release of a new standalone differential evolution sampler, Diver, and describe its design, usage and interface to ScannerBit. We subject Diver and three other samplers (the nested sampler MultiNest, the MCMC GreAT, and the native ScannerBit implementation of the ensemble Monte Carlo algorithm T-Walk) to a battery of statistical tests. For this we use a realistic physical likelihood function, based on the scalar singlet model of dark matter. We examine the performance of each sampler as a function of its adjustable settings, and the dimensionality of the sampling problem. We evaluate performance on four metrics: optimality of the best fit found, completeness in exploring the best-fit region, number of likelihood evaluations, and total runtime. For Bayesian posterior estimation at high resolution, T-Walk provides the most accurate and timely mapping of the full parameter space. For profile likelihood analysis in less than about ten dimensions, we find that Diver and MultiNest score similarly in terms of best fit and speed, outperforming GreAT and T-Walk; in ten or more dimensions, Diver substantially outperforms the other three samplers on all metrics.
Article PDF
Similar content being viewed by others
References
C.F. Berger, J.S. Gainer, J.A.L. Hewett, T.G. Rizzo, Supersymmetry without prejudice. JHEP 2, 23 (2009). arXiv:0812.0980
ATLAS Collaboration: ATLAS Collaboration, Summary of the ATLAS experiment’s sensitivity to supersymmetry after LHC Run 1—interpreted in the phenomenological MSSM. JHEP 10, 134 (2015). arXiv:1508.06608
N. Christensen, R. Meyer, L. Knox, B. Luey, Bayesian methods for cosmological parameter estimation from cosmic microwave background measurements. Class. Quantum Gravity 18, 2677–2688 (2001). arXiv:astro-ph/0103134
J. Dunkley, M. Bucher, P.G. Ferreira, K. Moodley, C. Skordis, Fast and reliable Markov chain Monte Carlo technique for cosmological parameter estimation. MNRAS 356, 925–936 (2005). arXiv:astro-ph/0405462
A. Lewis, S. Bridle, Cosmological parameters from CMB and other data: a Monte Carlo approach. Phys. Rev. D 66, 103511 (2002). arXiv:astro-ph/0205436
A. Lewis, S. Bridle, CosmoMC++, unpublished note (2006). http://cosmologist.info/notes/CosmoMC.pdf
E.A. Baltz, P. Gondolo, Markov Chain Monte Carlo exploration of minimal supergravity with implications for dark matter. JHEP 10, 52 (2004). arXiv:hep-ph/0407039
B.C. Allanach, C.G. Lester, Multidimensional mSUGRA likelihood maps. Phys. Rev. D 73, 015013 (2006). arXiv:hep-ph/0507283
P. Bechtle, K. Desch, P. Wienemann, Fittino, a program for determining MSSM parameters from collider observables using an iterative method. Comp. Phys. Commun. 174, 47–70 (2006). arXiv:hep-ph/0412012
R. Ruiz de Austri, R. Trotta, L. Roszkowski, A Markov chain Monte Carlo analysis of CMSSM. JHEP 5, 2 (2006). arXiv:hep-ph/0602028
O. Buchmueller, R. Cavanaugh et al., Predictions for supersymmetric particle masses using indirect experimental and cosmological constraints. JHEP 9, 117 (2008). arXiv:0808.4128
J. Skilling, Nested Sampling, in American Institute of Physics Conference Series, vol. 735, ed. by R. Fischer, R. Preuss, U.V. Toussaint, pp. 395–405 (2004)
R. Trotta, F. Feroz, M. Hobson, L. Roszkowski, R. Ruiz de Austri, The impact of priors and observables on parameter inferences in the constrained MSSM. JHEP 12, 24 (2008). arXiv:0809.3792
P. Scott, J. Conrad et al., Direct constraints on minimal supersymmetry from Fermi-LAT observations of the dwarf galaxy Segue 1. JCAP 1, 31 (2010). arXiv:0909.3300
Planck Collaboration, P.A.R. Ade, et. al., Planck 2015 results. XIII. Cosmological parameters. A&A 594, A13 (2016). arXiv:1502.01589
K.J. de Vries, E.A. Bagnaschi et al., The pMSSM10 after LHC run 1. Eur. Phys. J. C 75, 422 (2015). arXiv:1504.03260
F. Feroz, M.P. Hobson, M. Bridges, MULTINEST: an efficient and robust Bayesian inference tool for cosmology and particle physics. MNRAS 398, 1601–1614 (2009). arXiv:0809.3437
IceCube Collaboration, M. G. Aartsen et. al., Improved limits on dark matter annihilation in the Sun with the 79-string IceCube detector and implications for supersymmetry. JCAP 04, 022 (2016). arXiv:1601.00653
GAMBIT Collider Workgroup: C. Balázs, A. Buckley, et. al., ColliderBit: a GAMBIT module for the calculation of high-energy collider observables and likelihoods. Eur. Phys. J. C (in press) (2017). arXiv:1705.07919
Y. Akrami, P. Scott, J. Edsjö, J. Conrad, L. Bergström, A profile likelihood analysis of the constrained MSSM with genetic algorithms. JHEP 4, 57 (2010). arXiv:0910.3950
M. Ghulam, A. Faisal, M. Bilal, Optimization of neutrino oscillation parameters using differential evolution. Commun. Theor. Phys. 59, 324–330 (2013). arXiv:1109.2431
F. Feroz, K. Cranmer, M. Hobson, R. Ruiz de Austri, R. Trotta, Challenges of profile likelihood evaluation in multi-dimensional SUSY scans. JHEP 6, 42 (2011). arXiv:1101.3296
Y. Akrami, C. Savage, P. Scott, J. Conrad, J. Edsjö, Statistical coverage for supersymmetric parameter estimation: a case study with direct detection of dark matter. JCAP 7, 2 (2011). arXiv:1011.4297
M. Bridges, K. Cranmer et al., A coverage study of CMSSM based on ATLAS sensitivity using fast neural networks techniques. JHEP 3, 12 (2011). arXiv:1011.4306
C. Strege, R. Trotta, G. Bertone, A.H.G. Peter, P. Scott, Fundamental statistical limitations of future dark matter direct detection experiments. Phys. Rev. D 86, 023507 (2012). arXiv:1201.3631
P. Bechtle, J.E. Camargo-Molina et al., Killing the cMSSM softly. Eur. Phys. J. C 76, 96 (2016). arXiv:1508.05951
A. Putze, L. Derome, The Grenoble Analysis Toolkit (GreAT)—a statistical analysis framework. Phys. Dark Univ. 5, 29–34 (2014)
E .E .O. Ishida, S .D .P. Vitenti et al., COSMOABC: likelihood-free inference via population Monte Carlo approximate Bayesian computation. Astron. Comput. 13, 1–11 (2015). arXiv:1504.06129
GAMBIT Collaboration: P. Athron, C. Balazs, et. al., GAMBIT: the global and modular beyond-the-standard-model inference tool. arXiv:1705.07908
GAMBIT Dark Matter Workgroup: T. Bringmann, J. Conrad, et. al., DarkBit: a GAMBIT module for computing dark matter observables and likelihoods. arXiv:1705.07920
GAMBIT Models Workgroup: P. Athron, C. Balázs, et. al., SpecBit, DecayBit and PrecisionBit: GAMBIT modules for computing mass spectra, particle decay rates and precision observables. arXiv:1705.07936
GAMBIT Flavour Workgroup: F. U. Bernlochner, M. Chrzaszcz, et. al., FlavBit: a GAMBIT module for computing flavour observables and likelihoods. arXiv:1705.07933
GAMBIT Collaboration: P. Athron, C. Balázs, et. al., Global fits of GUT-scale SUSY models with GAMBIT. arXiv:1705.07935
GAMBIT Collaboration: P. Athron, C. Balázs, et. al., A global fit of the MSSM with GAMBIT. arXiv:1705.07917
GAMBIT Collaboration: P. Athron, C. Balázs, et. al., Status of the scalar singlet dark matter model. arXiv:1705.07931
P. Scott, Pippi—painless parsing, post-processing and plotting of posterior and likelihood samples. Eur. Phys. J. Plus 127, 138 (2012). arXiv:1206.2245
N. Metropolis, A.W. Rosenbluth, M.N. Rosenbluth, A.H. Teller, E. Teller, Equation of state calculations by fast computing machines. J. Chem. Phys. 21, 1087–1092 (1953)
D. MacKay, Information theory, inference, and learning algorithms (Cambridge University Press, Cambridge, 2003). (ISBN:0521642981)
R.M. Neal, Probabilistic inference using Markov Chain Monte Carlo methods. Technical Report CRG-TR-93-1 (1993)
W.K. Hastings, Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57, 97–109 (1970)
J.A. Christen, J. Weare, A general purpose sampling algorithm for continuous distributions (the t-walk). Bayesian Anal. 5, 263 (2010)
J. Goodman, J. Weare, Ensemble samplers with affine invariance. Commun. App. Math. Comput. Sci. 5, 65 (2010)
A. Gelman, D.B. Rubin, Inference from iterative simulation using multiple sequences. Stat. Sci. 7, 457–472 (1992)
R. Storn, K. Price, Differential evolution: a simple and efficient heuristic for global optimization over continuous spaces. J. Glob. Optim. 11, 341–359 (1997)
K. Price, R.M. Storn, J.A. Lampinen, Differential evolution: a practical approach to global optimization (Springer, Berlin, 2005)
S. Das, P. Suganthan, Differential evolution: a survey of the state-of-the-art. Evolut. Comput. IEEE Trans. 15, 4–31 (2011)
K. Price, Differential evolution, in Handbook of Optimization. Intelligent Systems Reference Library, vol. 38, ed. by I. Zelinka, V. Snášel, A. Abraham (Springer, Berlin, 2013), pp. 187–214
K. Price, R.M. Storn, J.A. Lampinen, The differential evolution algorithm, in Differential Evolution: A Practical Approach to Global Optimization, Natural Computing Series (Springer, Berlin, 2005), pp. 37–134
D. Zaharie, A comparative analysis of crossover variants in differential evolution. Proc. IMCSIT 2007, 171–181 (2007)
D. Zaharie, Statistical properties of differential evolution and related random search algorithms, in COMPSTAT 2008, ed. by P. Brito (Physica-Verlag, Heidelberg, 2008), pp. 473–485
E. Mezura-Montes, J. Velázquez-Reyes, C.A. Coello Coello, A comparative study of differential evolution variants for global optimization, in Proceedings of the 8th annual conference on Genetic and evolutionary computation, GECCO ’06 (ACM, New York, 2006), pp. 485–492
D. Zaharie, Influence of crossover on the behavior of differential evolution algorithms. Appl. Soft Comput. 9, 1126–1138 (2009)
J. Brest, S. Greiner, B. Boskovic, M. Mernik, V. Zumer, Self-adapting control parameters in differential evolution: a comparative study on numerical benchmark problems. Evolut. Comput. IEEE Trans. 10, 646–657 (2006)
F. Neri, V. Tirronen, Recent advances in differential evolution: a survey and experimental analysis. Artif. Intell. Rev. 33, 61–106 (2010)
A. Cuoco, B. Eiteneuer, J. Heisig, M. Krämer, A global fit of the \(\gamma \)-ray galactic center excess within the scalar singlet Higgs portal model. JCAP 6, 050 (2016). arXiv:1603.08228
A. Beniwal, F. Rajec et al., Combined analysis of effective Higgs portal dark matter models. Phys. Rev. D 93, 115016 (2016). arXiv:1512.06458
J.M. Cline, K. Kainulainen, P. Scott, C. Weniger, Update on scalar singlet dark matter. Phys. Rev. D 88, 055025 (2013). arXiv:1306.4710
K. Cheung, Y.-L.S. Tsai, P.-Y. Tseng, T.-C. Yuan, A. Zee, Global study of the simplest scalar phantom dark matter model. JCAP 1210, 042 (2012). arXiv:1207.4930
Y. Mambrini, Higgs searches and singlet scalar dark matter: combined constraints from XENON 100 and the LHC. Phys. Rev. D 84, 115017 (2011). arXiv:1108.0671
C.P. Burgess, M. Pospelov, T. ter Veldhuis, The minimal model of nonbaryonic dark matter: a singlet scalar. Nucl. Phys. B 619, 709–728 (2001). arXiv:hep-ph/0011335
J. McDonald, Gauge singlet scalars as cold dark matter. Phys. Rev. D 50, 3637–3649 (1994). arXiv:hep-ph/0702143
V. Silveira, A. Zee, Scalar phantoms. Phys. Lett. B 161, 136–140 (1985)
Particle Data Group: K. A. Olive et al, Review of Particle Physics. Chin. Phys. C 38, 090001 (2014)
Particle Data Group: K.A. Olive et. al., Review of Particle Physics, update to Ref. [62] (2015). http://pdg.lbl.gov/2015/tables/rpp2015-sum-gauge-higgs-bosons.pdf
M. Cacciari, G.P. Salam, G. Soyez, FastJet user manual. Eur. Phys. J. C 72, 1896 (2012). arXiv:1111.6097
P. Athron, J.-H. Park, D. Stöckinger, A. Voigt, FlexibleSUSY—a spectrum generator for supersymmetric models. Comput. Phys. Commun. 190, 139–172 (2015). arXiv:1406.2319
B.C. Allanach, SOFTSUSY: a program for calculating supersymmetric spectra. Comput. Phys. Commun. 143, 305–331 (2002). arXiv:hep-ph/0104145
Author information
Authors and Affiliations
Consortia
Corresponding author
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Funded by SCOAP3
About this article
Cite this article
The GAMBIT Scanner Workgroup: ., Martinez, G.D., McKay, J. et al. Comparison of statistical sampling methods with ScannerBit, the GAMBIT scanning module. Eur. Phys. J. C 77, 761 (2017). https://doi.org/10.1140/epjc/s10052-017-5274-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1140/epjc/s10052-017-5274-y