Workflows for Ultra-High Resolution 3D Models of the Human Brain on Massively Parallel Supercomputers

  • Hartmut MohlbergEmail author
  • Bastian Tweddell
  • Thomas Lippert
  • Katrin Amunts
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10087)


Human brain atlases [1] are indispensable tools to achieve a better understanding of the multilevel organization of the brain through integrating and analyzing data from different brains, sources, and modalities while considering the functionally relevant topography of the brain [4]. The spatial resolution of most of these electronic atlases is in the range of millimeters, which does not allow the integration of the information at the level of cortical layers, columns, microcircuits or cells. Therefore, we introduced in 2013 the first BigBrain data set with a resolution of 20 \(\upmu \)m isotropic. This data set allows to specify morphometric parameters of human brain organization, which serve as a “gold standard” for neuroimaging data obtained at a lower spatial resolution. It provides, in addition, an essential basis for realistic brain models concerning structural analysis and simulation [2]. For the generation of other, even higher-resolution data sets of the human brain, we developed an improved and more efficient data processing workflow employing high performance computing to 3D reconstruct histological data sets. To facilitate the analysis of intersubject variability on a microscopic level, the new processing framework was applied for reconstructing a second BigBrain data set with 7676 sections. Efficient data processing of a large amount of data sets with a complex nested reconstruction workflow using large number of compute nodes required optimized distributed processing workflows as well as parallel programming. A detailed documentation of the processing steps and the complex inter-dependencies of the data sets at each level of the multi-step reconstruction workflow was essential to enable transformations to images of the same histological sections obtained with even higher spatial resolution. We have addressed these challenges, and achieved efficient high throughput processing of thousands of images of histological sections in combination with sufficient flexibility, based on an effective, successive coarse-to-fine hierarchical processing.


Ultra-high resolution brain models BigBrain Cytoarchitecture Microstructure High performance computing Workflows Human Brain Atlas 



The authors thank Claude Lepage and Alan C. Evans from the Montreal Neurological Institute, Montreal, Canada, for developing and providing the HPC workflow for the 3-D reconstruction of the first BigBrain, which served as a basis for the recent workflow and for many fruitful and inspiring discussions. We thank Ana Oros-Peusquens and Nadim J. Shah for their contribution in MR-imaging. Funding was provided for this study by the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 604102 (Human Brain Project), and the Portfolio Theme Supercomputing and Modeling for the Human Brain of the German Helmholtz Association.


  1. 1.
    Zilles, K., Amunts, K.: Centenary of Brodmann’s map - conception and fate. Nat. Rev. Neurosci. 11(2), 139–145 (2010)CrossRefGoogle Scholar
  2. 2.
    Amunts, K., Lepage, C., Borgeat, L., Mohlberg, H., Dickscheid, T., Rousseau, M., Bludau, S., Bazin, P., Lewis, L., Oros-Peusquens, A., Shah, N., Lippert, T., Zilles, K., Evans, A.C.: BigBrain: an ultrahigh-resolution 3D human brain model. Science 340, 1472–1475 (2013)CrossRefGoogle Scholar
  3. 3.
    Amunts, K., Bücker, O., Axer, M.: Towards a multiscale, high-resolution model of the human brain. In: Grandinetti, L., Lippert, T., Petkov, N. (eds.) BrainComp 2013. LNCS, vol. 8603, pp. 3–14. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-12084-3_1 Google Scholar
  4. 4.
    Amunts, K., Hawrylycz, M., Van Essen, D., Van Horn, J.D., Harel, N., Poline, J.B., De Martino, F., Bjaalie, J.G., Dehaene-Lambertz, G., Dehaene, S., Valdes-Sosa, P., Thirion, B., Zilles, K., Hill, S.L., Abrams, M.B., Tass, P.A., Vanduffel, W., Evans, A.C., Eickhoff, S.B.: Interoperable atlases of the human brain. Neuroimage 99, 525–532 (2014)CrossRefGoogle Scholar
  5. 5.
    Brodmann, K.: Vergleichende Lokalisationslehre der Großhirnrinde in ihren Prinzipien dargestellt auf Grund des Zellbaues. Barth, Leipzig (1909)Google Scholar
  6. 6.
    Amunts, K., Malikovic, A., Mohlberg, H., Schormann, T., Zilles, K.: Brodmanns area 17 and 18 brought into stereotaxic space where and how variable? Neuroimage 11(1), 66–84 (2000)CrossRefGoogle Scholar
  7. 7.
    Eickhoff, S., Stephan, K.E., Mohlberg, H., Grefkes, C., Fink, G.R., Amunts, K., Zilles, K.: A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage 25(4), 1325–1335 (2005)CrossRefGoogle Scholar
  8. 8.
    Evans, A.C., Janke, A.L., Collins, D.L., Bailllet, S.: Brain templates and atlases. Neuroimage 62(2), 911–922 (2012)CrossRefGoogle Scholar
  9. 9.
    Delescluse, M., Franconville, R., Joucla, S., Lieurya, T., Pouzat, C.: Making neurophysiological data analysis reproducible. Why and how? J. Physiol. Paris 106, 159–170 (2011)CrossRefGoogle Scholar
  10. 10.
    Hömke, L.: A multigrid method for anisotropic PDEs in elastic image registration. Numer. Linear Algebra Appl. 13, 215–229 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Lepage, C., Mohlberg, H., Pietrzyk, U., Amunts, K., Zilles, K., Evans, A.C.: Automatic repair of acquisition defects in reconstruction of histology slices of the human brain. In: 16th Annual Meeting of the Organization for Human Brain Mapping (OHBM), Barcelona, 06. - 10.06.2010: available on CD-ROM (2010)Google Scholar
  12. 12.
    Zhu, C., Byrd, R.H., Nocedal, L.: L-BFGS-B: Algorithm 778: L-BFGS-B, FORTRAN routines for large scale bound constrained optimization. ACM Trans. Math. Softw. 23(4), 550–560 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Thévenaz, P., Blu, T., Unser, M.: Interpolation revisited. IEEE Trans. Med. Imaging 19(7), 739–758 (2000)CrossRefGoogle Scholar
  14. 14.
    Lewis, L., Lepage, C., Fournier, M., Zilles, K., Amunts, K., Evans, A.C.: BigBrain: Initial tissue classification and surface extraction. In: 20th Annual Meeting of the Organization for Human Brain Mapping (OHBM), Hamburg (2014)Google Scholar
  15. 15.
    Liu, S., Azevedo, C., Pelletier, D.: The use of BigBrain in MS: An ultrahigh-resolution 3D template for grey matter MRI segmentation. Neurology 82(10), 6.135 (2014)Google Scholar
  16. 16.
    Wagstyl, K., Lepage, C., Zilles, K., Amunts, K., Fletcher, P., Evans, A.: BigBrain: Automated analysis of laminar structure in the cerebral cortex. In: 22nd Annual Meeting of the Organization for Human Brain Mapping (OHBM), Geneva (2016)Google Scholar
  17. 17.
  18. 18.
    Python Software Foundation: Python Language Reference, Version 2.7.
  19. 19.
  20. 20.
    Graph-tool: Efficient network analysis with Python.
  21. 21.
    Insight Segmentation and Registration Toolkit (ITK).
  22. 22.
    The HDF Group: Hierarchical data format (HDF), Version 5.
  23. 23.
  24. 24.
    The Open Provenance Model (OPM).
  25. 25.
    The Human Brain Project (HBP).

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Hartmut Mohlberg
    • 1
    Email author
  • Bastian Tweddell
    • 2
  • Thomas Lippert
    • 2
  • Katrin Amunts
    • 1
    • 3
  1. 1.Institute of Neuroscience and Medicine (INM-1), Research Centre Juelich and JARA-BrainJülichGermany
  2. 2.Juelich Supercomputing Centre (JSC), IAS, and JARA-HPC, Research Centre JuelichJülichGermany
  3. 3.C. and O. Vogt Institute of Brain ResearchHeinrich-Heine UniversityDüsseldorfGermany

Personalised recommendations